neptune AI

What do you like best about neptune.ai?

I've found Neptune to be a flexible, easy to use platform for tracking everything in our ML development process. The API is simple to set up and requires minimal code to track experiments. I've been loving the runs table, it makes it easy to group and filter experiments for quick comparisons.

Their support team has been great at responding to any questions and showing us new features as they are released.

What do you dislike about neptune.ai?

Dashboard visualizations automatically resize when you adjust the window size, meaning you need to resize and reorder the visualizations at that size. Would love if there was a way to fix the size of dashboards so this wouldn't happen.

What problems is neptune.ai solving and how is that benefiting you?

Dataset tracking: Allows us to track our detailed EDAs everytime we run them, allowing us to see what's going on when datasets change

Experiment tracking: Integrates with Optuna for hyperparameter tuning, allowing us to easily share experiments betyween team members. The runs table makes it easy to compare different optuna experiments.

Logs and memory usage tracking in production: We've found the neptune UI makes viewing memory usage and logs from our production runs rather than aws log views, making monitoring more accessible to the team.

Test and production model tracking: Gives us the tools to get deep insights into every model run, and share them amoung the team.

What do you like best about neptune.ai?

Neptune Ai, allows to store practically any necessary data, based on its way of storing data and metadata, it allows complete traceability in a simple way.

It has an easy to use UI, which exposes, in a simple and useful way, practically all the necessary information, allowing some interesting customisations, such as being able to add/remove custom visualisations only with the necessary data/columns.

It also has a Python SDK which is quite easy to use, allowing to adapt to the use case in a quite flexible way, as Neptune provides an accessible data and metadata container, so integration with other tools is quite feasible.

On the support side, they are usually quite quick to respond and provide a lot of help. Another very interesting point is that the documentation is quite extensive and has a lot of examples.

What do you dislike about neptune.ai?

The great flexibility that Neptune AI provides as a container of data and metadata is both an advantage and a disadvantage, as it requires a good level of governance not to record too much information, or organise it in the right way, to be really useful. This is not a ‘problem’ of the tool itself, but it is something to be taken into account when using it in a use case of some complexity.

What problems is neptune.ai solving and how is that benefiting you?

Model registry of ML models and metadata storage

What do you like best about neptune.ai?

The ease of use is super, just a couple of lines of code, and you are ready to track and log the metrics you want. Documentation and customer support are super efficient, and you can uploads tons of metrics. After a couple of connection issues (solved easily) we were able to integrate from different pipelines on azureML studio.

What do you dislike about neptune.ai?

We experienced some problems with connection stability and timeouts while uploading metrics, but everything was solved easily after a fast communication with customer support.

Also a slightly higher focus on LLM evaluation would be appreciated especially in these LLMs-centric days.

What problems is neptune.ai solving and how is that benefiting you?

It is making the ML experiments tracking way more efficient and fast. Forget excel tables, manual comparison and wasted time.

What do you like best about neptune.ai?

Neptune just works for tracking experiments: it has nice and not overloaded UI that is quite fast (especially in the new version), you can track metrics and compare different runs. The team also provides great support and open to implementing new features on request. We use it for tracking all of our pretraining, post-training and evaluations.

What do you dislike about neptune.ai?

When you want to compare just final metrics of multiple runs as a table, that's something Neptune can do but the experience can be improved.

Also some features important to us are still missing, like being able to log steps of one run non-monotonically from different processes.

What problems is neptune.ai solving and how is that benefiting you?

Tracking metrics of our LLM pretraining, post-training and evaluations

What do you like best about neptune.ai?

The fact that I cane keep track of my experiments easily:

- They are all on one place

- I can compare between runs

- I can log nearly everything I want even when tuning with optuna I can see all the nice metadata and visulizations.

-

What do you dislike about neptune.ai?

A few minor things:

- The fact that everytime I visit the page of my projects the colors for each run are different. There is an option to change the color but I do not want to have to change the color of each run especially when I am tuning a model which results in hundreds of runs.

- The stderr and stdout logs are unreadable especially when I am using progress bars.

- The optuna visualizations in full screen mode are quite small, especially the contours. I would expect that entering a full screen mode they would adjust according to my screen size and resolution.

What problems is neptune.ai solving and how is that benefiting you?

Keeping my experiments in one place. It is easy to revisit and inspect what models I have trained, with what hyper-parameters, how they perform, etc. This way it is very easy to go for a v2 of the models without missing a thing.