Subsequent Treatment

[rsenabre]

Hi! I was wondering if there is a way on the CL Db to know if a case was reversed or narrowed by other case. I was trying to find it on the tables but didn't succeded.

[mlissner]

Unfortunately, no. It's been on our want list for a long time, but we haven't been able to get the resources to do it even in a basic way.

[legaltextai]

i 've been thinking how to implement this idea too, just some thoughts:

1. if we are sure "Cited By" contains all the cases citing this opinion -> get AI to categorize the treatment in real time (may take some time depending on the number of citing cases);

2. convert all published opinions using a pre-trained , properly prompted AI model into a json format which would also include:
   Cases cited (which cases the court cited and how it treated them):

   • 'red flag' means some negative treatment — such as a cited case is being overruled, superseded, or not followed by this court for some reason.
   • 'yellow flag' means the case has some negative treatment by this court but has not been reversed or overruled. For example, the reasoning of the decision was criticized or its holding was limited to a specific set of facts.
   • 'blue flag' means the case has been appealed to the U.S. Court of Appeals or the U.S. Supreme Court (excluding appeals originating from agencies).
     -> then search by case name / citation (much faster than the first option) to see the treatment.

   I put here a quick prototype to show a possible way to convert an opinion summary into a json format, plus treatment of cited cases. Give it some time to load as it is on a free streamlit cloud. Try 'brown v board' .

[rsenabre]

I like the idea, but I have some doubts about scalability and related costs.

• If the idea is to evaluate all the cases, it will take much processing time and model cost.
• Suppose that is solved, there are some cases like Roe v Wade that has thousands of citations and was overruled on 2022.

I was thinking on creating some kind of metric like citations per year to address relevance of the citing case and with graphs or another structure, find those cases that are more relevant than the cited case and then apply AI to compare

[mlissner]

That's a cool demo. I agree with everything here. The cost could be a problem, and there's also value to doing some basic citation trendlines on more-frequently cited cases to look for dropoffs or things like that (I think people would be into this regardless!).

I'll also add that some basic heuristics are probably quite doable using things like regex or basic ML.

[legaltextai]

I would focus on the published opinions with > 200 words only.
For thousands of citations problem, why not to limit to the last say 10 by date.
Summarizing 5-6mln opinions should not take that long. You do it once and then just run for new updates only.
Should not take longer than embedding full texts which I 've done here

[legaltextai]

Are you using SERP API for case search? The last time I checked, they did not have case search yet. Depending on your use, it may get costly. Why not use CL data, create a postgres db and do a keyword/semantic search over specific fields?
Make sure you split the text according to the embedding model size (e.g. 512 tokens, with some overlap).
Re-joining the chunks adds more time, no?
Trying with very large embedding models like SFR-Embedding-Mistral that can embed 32K at once will be interesting. Let me know please how it worked for you.

[rsenabre]

Yes, using SerpAPi for that, they allowed case search and the cost is not massive.
For the courtlistener data we're storing on a MySql db and making embeddings of course, I think the 512 tokens window won't be good enough, so I'm trying to build something bigger.

About rejoining the chunks, no, with pinecone is fairly easy.

[legaltextai]

Does not Pinecone have a limit on how much you can stuff into metadata field? I remember smth like 40kb.
I may sound like a broken record, take a look at a combination of postgres + pgvector (version 0.6)

[mlissner]

Just here to reiterate that if we do this for CL itself, we'll probably want to avoid pinecone, since it's proprietary. (But if you are doing this in your own systems, don't let me stop you, obviously!)

[legaltextai]

I was responding to @rsenabre who mentioned pinecone.

[mlissner]

This definitely deserves some more effort, but I think it would be an incredible addition to our system if there was energy to build it. I wonder if this could fit into eyecite as some sort of add-on, maybe.

The other thing I'll add is that we don't have all the citations, and sometimes we add ones we previously lacked. That makes things tricky, but, OK, we can probably deal with that!

[legaltextai]

I would highly encourage you to consider introducing semantic/hybrid search capabilities on CL. Integrating an eyecite/treatment add-on could be a part of that endeavor too. The necessary tools are available, which could transform CL into a playground for experiments with legal data. What do you perceive as the main bottleneck? Is it server costs, embedding costs, or something else?

[rsenabre]

Yes! We're doing that,I was wrestling with the embeddings cost, because I made a rough (extremely rough) estimation with a 5% of the database and it will cost around $2500 to embed it using openai. I'm digging into another possibilities as we discussed before. The bottleneck is the subsequent treatment.

[mlissner]

We have an issue for vector search: https://github.com/freelawproject/courtlistener/issues/3398. The main bottleneck is somebody to get it done, but the cost of embeddings worries me a good deal too.

[legaltextai]

There are free embedding models. If you mean vector storage costs, providers like Weaviate and Pinecone would be super expensive given the amount of data. You will need a postgres database hosted on a server with 1-2 TB SSD. Probably $50 per month. Plus, the costs of Cohere Rerank if you would like to add that to make search results even better.

[mlissner]

So we're using Elastic now and it supports some of this. Storage is fine. It's computing the embeddings that I worry about, but I think the human cost of this is probably the biggest one. In general, we avoid using co's like Pinecone, Weaviate, Cohere, etc, if we can. Open approaches get a strong preference around here.

[legaltextai]

My biggest time spend when working with CL data has involved joining the necessary tables and trimming off unnecessary data. The embedding process can be put on auto-pilot.

[legaltextai]

[legaltextai]

Look at that initial record upload. That's steep.

[rsenabre]

maybe 48m is too much, if you lower to 11m then it's something more reasonable

[rsenabre]

I will check the pgvector solution anyway and I'll let you know how it works with Mistral

[legaltextai]

from my experience, if you use 1024 vector embedding model / 512 token context, each decision will need to be split on average into 5-6 chunks. 8 mln decisions x 6 = 48 mln

[rsenabre]

By using text-embedding-ada-002 or a bigger context model we estimate we could downsize it to around 12 millon records after cleaning

[legaltextai]

I will check the pgvector solution anyway and I'll let you know how it works with Mistral

feel free to email me (admin at legaltext dot ai) if you need help with pg implementation.