Has anyone experiment with using VLM to detect "marks"? Thinking of pen/pencil based markings like underlines, circles,checkmarks.. Can these models do it?

None of them do it well from our experience. We had to write our own custom pipeline with a mixture of legacy CV approaches to handle this (AI contract analysis). We constantly benchmark every new multimodal and VLM model that comes out and are consistently disappointed.

If someone releases a benchmark/dataset, I'm sure that significantly increases the chances of one of these AI labs training on the task.

Would this work by just feeding it my obsidian vault?

Yep - if you just want the assistant to access your Obsidian vault, you can point it to the vault during chat. If you’d like it to show up in the UI as well, or don't want to point to it on every chat, you can copy the vault to a folder under knowledge: ~/.rowboat/knowledge/obsidian-vault.

Anyone use these approaches with academic pdfs?

Another approach is to teach Claude Code how to use your Zotero library's full-text search: https://github.com/urschrei/zotero_search_skill.

I've not seen any impressive products. But products do exist ie https://scibite.com/solutions/semantic-search/

Anyone using them for electronics datasheets?

I would like to. I haven't yet found a solution that works well.

The problems with datasheets is tables which span multiple pages, embedded images for diagrams and plots, they're generally PDFs, and only sometimes are they 2-column layout.

Converting from PDF to markdown while retaining tables correctly seems to work well for me with Mistral's latest OCR model, but this isn't an open model. Using docling with different models has produced much worse results.

I've been working on a tool specifically to handle these messy PDF-to-Markdown conversions because I ran into the same issues with tables and multi-column layouts.

I’ve optimized https://markdownconverter.pro/pdf-to-markdown to handle complex PDFs, including those tricky tables that span multiple pages and 2-column formats that usually trip up tools like Docling. It also extracts embedded diagrams/images and links them properly in the output.

Full disclosure: I'm the developer behind it. I’d love to see if it handles your specific datasheets better than the models you've tried. Feel free to give it a spin!

Cool! But given that often electronics documentation is covered by NDAs, my preferred solution is local-first if at all possible.