Update April 29, 2024
🎨 Visual Editor
It's been four busy weeks -- we were so engrossed in making Breadboard awesome that we missed one update. This one will have to make up for it.
One of the big updates this week is that Breadboard Visual Editor is functionally on par with the code-first workflow 🎉. This is a huge milestone for the project: you can now build all graphs with Visual Editor that you can build with code.
Ever since we started on the editor in late January, it's been a mad dash to get all the bits in place. And finally, we're at a point where all the underlying physics of the Breadboard Graph Language (BGL) are expressible with the Visual Editor.
A lot of work went into this.
Embedded graphs
We helped the Visual Editor learn about embedded graphs. Breadboard allows embedding graphs within graphs. This can be very handy for breaking up one large board into chunks, while still keeping them all in the same BGL file. The embedded graphs are also typically used by map and reduce nodes as functional-programming style arguments.
In addition to creating new embedded graphs, we also taught the Editor to let the user pick them when configuring nodes with a UI (as opposed to typing in the graph URLs):
Wire kinds
We taught the Editor about the four different kinds of wires that can be used to connect the nodes: regular, constant, control, and "star".
- Regular wires connect one output port of a node to an input port of another (or the same) node.
- Constant wires are like regular, except they memoize the port value and make it always available for the respective input port.
- "Star" wires take all output ports of a node and bring it as input ports to another node
- Control wires convey no data. They are only here for control flow.
Node metadata
The Visual Editor learned about node metadata and how to allow users to edit it. The node metadata is super-useful when working with larger graphs. Those titles and descriptions go a long way in helping you and the next person who looks at a graph to make sense of it.
Improved schema editor
We also taught the Visual Editor about the richness of the schema that helps express inputs and outputs of a graph. From types to formats to defaults and examples, to behavior hints: all of these are now accessible with the Visual Editor.
Content modalities and input types
Along the way, we helped Editor learn to handle a wider variety of data types in addition to text, numbers, and JSON. For instance, here's an audio input where you can just hold down a button to record audio:
Improved node appearance
The Visual editor also grew additional controls for how a node appears in the editor. You can now double-click on a node to minimize it entirely, reducing clutter in complex graphs:
You can also access this feature and an option to delete this node via the overflow menu:
Settings
Also new is the settings window, where the you can manage various flags, secrets, and other bits in one place.
Code editor
Not that Visual Editor is jealous of the code-first approach, but when editing Javascript code in runJavascript, the Visual Editor will now do proper syntax highlighting. It doesn't want us to abandon our code-first ways. It just hints that there may be better, more visual ways.
Bug fixes and more polish
And of course, we spend inordinate amount of time sweating over the small stuff, polish, fixing bugs, and making sure that Breadboard Visual Editor is rock-solid.
🧑🍳 On the stove
...And it is still rapidly evolving. Our aim today is to teach the Visual Editor to be more intuitive and easier to use. While expressing the entirety of the BGL was a necessary first step, we are now looking to find ways to reduce the cognitive load for newcomers and add shortcuts for common tasks -- just like the full knowledge of Newtonian physics is not necessary for performing practical everyday tasks (despite all of them being governed by physics).
One idea we're playing with is the notion of a simplified representation, which elides many of the rarely-used features. For instance, we could go from:
To a more streamlined:
Additionally, we are exploring the idea of Visual Editor "presets": customizable views of a Visual Editor that are targeted a particular task. For instance, my preset for building asynchronous, semi-autonomous agents might be different from your preset for creating boards that invoke various APIs directly and perform other low-level operations.
Code-first syntax: next generation
In parallel, a refactoring effort is underway to convert all of the existing code-first boards to the next generation of syntax. We already have the Template Kit converted, and many of the tricky Core Kit nodes converted to this syntax, with more conversion underway.
Python notebooks in Breadboard
We're also building Jupyter support for Breadboard. In our early prototypes, we can already run Jupyter notebooks as graphs. This is a big deal, because it opens the path for a nearly seamless interaction between a notebook and a board.
Human feedback with Agent Kit
Another interesting area we're experimenting with is structured feedback. In the Agent Kit, the human node allows a way to insert a point of human interaction in a graph of synthetic workers. At the moment, the node only supports free text input (kind of like a typical chat). We are teaching it to support more structured inputs as well. For instance, for some interactions, it might just be enough to say "yes" or "no":
And then, if the user didn't like it, direct the worker to repeat the task:
LLM Content Editor
Last, but definitely not least, here's a sneak peek at the new multimodal content editor that we're developing. It enables uploading images, videos, sounds, PDFs, and more as part of the Visual Editor experience. Anytime there's an input into an LLM, the multimodal content editor will let you focus on building the best prompt, task, system instruction, or character -- and take care of importing, transforming, and assembling your work into proper formats.
Whew, that's all for this round of updates. See you in a couple of weeks!