Sharing because I was wondering if OpenAI would built a bot in the future and it seems to me having spacial audio on the model could provide a hint at that.
A small experiment to see if we are there yet with highly virtualized CPU compute and Small Language Models (SLM). The answer is a resounding maybe, but most likely not. Huge thanks to Justine for her work on Llamafile supported by Mozilla. Hope folks find this R&D useful.
Does it produce bad results? Is it slow to respond? Slow to load?
I've been wanting to play around with llamafile-based edge functions but storing even small models in GitHub (for automated deploys) is a terrible and often impossible experience.
Yup, this seems right. You pay for tokens no matter what. Even in other APIs. Did you know you can set an expire for files, vector stores, etc? No need to pay for long term storage on those. Also, threads are free.
Not OP, but I use TypeScript because it adds a layer of safety to the codebase.
It's like having good test coverage - you can make large changes and if the tests pass (the code compiles), you can be fairly confident that you didn't mess anything up.
I've written Ruby for years, so I'm used to dynamically typed languages. But JavaScript is it's own level of special, and there's so many ways you can accidentally mess things up.
Having tests cover every single path (especially failure paths) can be very time consuming, and often hard or messy to setup (how would you mock the OpenAI module returning an error when adding metadata to a thread?), where as using something like TypeScript can make sure your code handles all paths somewhat correctly (at least as well as the types you defined).
Your code looks clean, and you appear to have good test coverage, so you do you though :-)
Great points. Dont even get me started about how function calling in other LLMs costs me tokens. Something OpenAI provides OOTB. I'm also not a big fan of OpenAI's lock in. Right now I'm on a huge Claude 3 Haiku kick. That said, OpenAI does seem to get the APIs right and my hunch is the new Assistants API is going to potentially disrupt things again. Time will tell.
I would love to be using Claude, but you can't get API access (beyond an initial trial period) in the EU without providing a European VAT number. They don't want personal users or people to even learn and experiment I guess.
Opus is really cool. I’ve found it to have a few persistent bugs in what I initially assumed is tokenization but now wonder if might be more fundamental, but modulo a few typographical-level errors, I personally think it’s the most useful of the API-mediated models for involved sessions.
And there are some serious people at Anthropic, they’ll get the typo thing if they haven’t already (been a busy week and change, they easily could have shipped a fix and I overlooked it).
I have some assumptions/guesses on how billing works. Gonna do a post on this on my unremarkable.ai blog, please do signup for posts there, no spam. I could be right or wrong but need to do some experiments and publish later.
Thanks @beoberha, I am too. I like one take I heard on Twitter. The sentiment was something like these types of systems are useful under the AI-Powered Productivity industry which has incremental gains, no big bangs. Said another way, if your job was to help a TON of your employees be more productive individually, it is worth it because companies measure those efforts broadly and the payoff is there. But again, not big. My advice for folks to stay lower level and hook AI automation up with simple, closed loop, LLM patterns that feel more like basic API calls in a choreographed manner. OMG, hope all that made sense
Glad Google is catching up with Ruby. But if you really want to learn how I am doing this with AWS and how easy it is to handle Rails on Lambda, check out the Lamby work here. https://lamby.custominktech.com
Many of my users were friends I met at various Ruby/Rails development circles. HomeMarks is a passion project that I have strived to re-write every few years. I threw away ~7K lines of custom MV* JS (before there were any) written in Prototype.js to create this new version in Rails/SpineJS. I hope others find it useful. History here.
