Case Study: DaaX LAKEer vs. ChatGPT-5 – Oil & Gas Natural Language Search

Backstory of This Case Study

We at DaaX were discussing our technology with an AI technology group in a business unit of a leading oil and gas company. After several technical deep-dive meetings, they asked us to benchmark DaaX vs. a LLM. This said the benchmark results would help them reach internal consensus on the next steps with DaaX. To make this an honest benchmark we selected ChatGPT-5 as the LLM, as it's one of the best.

We at DaaX are AI technology experts, not oil & gas domain experts. We didn't know what to benchmark, what tests to run that would be meaningful and valuable for the customer.

So, we asked the customer for help in developing the benchmark test – and they agreed.

They identified a publicly-available document to search using natural language queries – an oil well completion report published by Santos and made publicly available by Queensland (Australia) government https://geoscience.data.qld.gov.au/data/report/cr106865. This document, over 100-pages, is extremely challenging to search. It has text, merged cells in tables, figures, and has a tremendous amount of domain-specific language.

And they gave us three natural language questions to use to search this document:

1. Summarize any Non-Productive Time (NPT) events, equipment failures, or Health, Safety, and Environmental (HSE) incidents that occurred during drilling, casing, or cementing operations. Include the type of event, date, affected equipment or personnel, and any corrective actions taken.
2. Identify any significant deviations between actual and prognosed depths for key stratigraphic markers encountered during drilling. Include the marker name, expected depth, actual depth, deviation magnitude, and any geological or operational explanations provided.
3. Provide a comparison of the surface and production holes of the well including depth, hole/bit size, drilling fluid details, and interval. Also, identify any notable mud losses, fluid-related operational issues, or mitigation actions taken during the operation of drill fluid program.

We were very happy with the assistance from the customer; without it this benchmark would never have occurred. We'd have struggled to find a suitable document to use and struggled even more to figure out questions to use to search the document.

Benchmark Methodology

DaaX setup our LAKEer agent with (1) the Santos oil well completion report, and (2) a small portion of the oil & gas industry-standard ontology at https://github.com/Accenture/OSDU-Ontology. LAKEer ingests unstructured data and extracts pertinent information to populate a 3D graph. The OSDU ontology is used to provide domain-specific understanding to the 3D graph. Why a small portion of the ontology? When we did this benchmark, adding ontologies to LAKEer required a fair bit of engineering and we didn't have enough time to ingest the entire ontology. Since then, LAKEer has progressed, so this time has greatly reduced.

We then searched the Santos document with the three natural language queries and captured the results.

For ChatGPT-5, we uploaded the Santos document and ran the following prompts and captured the results.

1. ChatGPT-5 prompt for question 1: For the attached document, answer the following question: Summarize any Non-Productive Time (NPT) events, equipment failures, or Health, Safety, and Environmental (HSE) incidents that occurred during drilling, casing, or cementing operations. Include the type of event, date, affected equipment or personnel, and any corrective actions taken
2. ChatGPT-5 prompt for question 2: Now answer this question: Identify any significant deviations between actual and prognosed depths for key stratigraphic markers encountered during drilling. Include the marker name, expected depth, actual depth, deviation magnitude, and any geological or operational explanations provided
3. ChatGPT-5 prompt for question 3: Now answer this question: Provide a comparison of the surface and production holes of the well including depth, hole/bit size, drilling fluid details, and interval. Also, identify any notable mud losses, fluid-related operational issues, or mitigation actions taken during the operation of drill fluid program

Benchmark Results

We ran the comparison and now what?

Again, our lack of domain knowledge presented a big challenge for us – how do we evaluate the results? Are we objectively better or worse than ChatGPT-5?

Here we just brute forced it – we went through the Santos document repeatedly. We read it over-and-over until we were able to answer the questions ourselves. It took us a long-time to do this, we struggled to understand the Santos document but eventually got it.

The output of us going through the document manually was our source of truth with which we could compare LAKEer and ChatGPT-5 answers against.

Question 1:

Summarize any Non-Productive Time (NPT) events, equipment failures, or Health, Safety, and Environmental (HSE) incidents that occurred during drilling, casing, or cementing operations. Include the type of event, date, affected equipment or personnel, and any corrective actions taken.

Result: Correct answer is 4-hours. Chat GPT incorrectly says 0-hours, DaaX LAKEer correctly says 4-hours

ChatGPT-5 Answer for Question 1

LAKEer Answer for Question 1

Question 2:

Identify any significant deviations between actual and prognosed depths for key stratigraphic markers encountered during drilling. Include the marker name, expected depth, actual depth, deviation magnitude, and any geological or operational explanations provided.

Result: Correct answer for Top of Walloon Coal Measures is there is no data. Chat GPT incorrectly says ≈ 515 m, DaaX LAKEer correctly says that Data not available

ChatGPT-5 Answer for Question 2

LAKEer Answer for Question 2

Question 3:

Provide a comparison of the surface and production holes of the well including depth, hole/bit size, drilling fluid details, and interval. Also, identify any notable mud losses, fluid-related operational issues, or mitigation actions taken during the operation of drill fluid program.

Result: Chat GPT answers are irrelevant and meandering, DaaX LAKEer answers are spot-on.

ChatGPT-5 Answer for Question 3

LAKEer Answer for Question 3

We believe these results demonstrate that DaaX LAKEer consistently outperformed ChatGPT-5 in the quality of answers.

Additional LAKEer Differentiation vs. ChatGPT-5

Complex Query Decomposition

LAKEer decomposes complex user queries into simpler sub-queries, retrieves targeted answers, and fuses the responses into a unified, coherent result. This approach significantly improves accuracy for multi-part technical questions.

For example, LAKEer decomposed query #1 (Summarize any Non-Productive Time (NPT) events, equipment failures, or Health, Safety, and Environmental (HSE) incidents that occurred during drilling, casing, or cementing operations. Include the type of event, date, affected equipment or personnel, and any corrective actions taken) into the following 3 sub-queries. It answered these 3 sub-queries and then fused the answers together into the final answer returned to the user.

Mathematical Computations

DaaX LAKEer extracts numeric values from tables and curves and stores them as true numerical values in a database. This enables accurate mathematical comparisons across wells and depths. LLMs typically handle numbers as text without structured numeric computation.

Conclusion

As mentioned above, we believe these results demonstrate that DaaX LAKEer consistently outperformed ChatGPT-5 in the quality of answers.

We believe that LLMs need a helping hand for Enterprise Search. DaaX provides that help by combining (1) 3D Graphs with LLMs (LAKEer used ChatGPT) (2) Adding in Domain Knowledge, and (3) Adding in Company Language – leading to the superior results shown in this case study.

You can try running these benchmarks yourself. You can download the Santos document, upload it to ChatGPT-5 and run the three questions listed above. And you can query our LAKEer energy demo, which already has the Santos document ingested, with the three benchmark questions.