https://www.trygloo.com/blog/classify-text-llms-learnings

Last Checked: Jul 12, 2023, 17:38 EDT

Check another URL
IP Address: 76.76.21.241
ASN #: AS16509 AMAZON-02, US
Location: Unknown, Unknown, Unknown
URL Reputation:
  • Unknown This URL is not identified as malicious in the PhishTank Database.
  • Unknown PhishCheck thinks this URL is likely not a phish.
  • Unknown OpenPhish: URL not in feed.

Other submissions on 76.76.21.241:

  • https://www.deepswap.ai/?utm_source=exo&cp_id=4838322&ck_id=ooc4QfnU21S3TyurutqstlldNPVTZK5001rpp7HTumrdK6V0rqKaJbrLrHUUUWVSyuoltonsmdK6Z0rpXSuldM6V0rpnOdK7nNcu2a49Q_uc6V0rpXSuldK6V0rpXB9g

    Details

  • https://www.travelsupport-ca.com/

    Details

  • https://jemi.so/questconsumablescom

    Details

  • https://porncrafty.com/

    Details

  • https://star.gift/

    Details

  • https://www.mizzenandmain.com/

    Details

  • https://mint-mysterybox-claimz012.vercel.app/

    Details

  • https://verifys-validates-connect.vercel.app/

    Details

  • https://mysteryclaims29g.vercel.app/

    Details

  • https://mysteryclicks8g.vercel.app/

    Details

Other submissions on trygloo.com:

Previous checks:

                               Domain Name: trygloo.com
Registry Domain ID: 2761163473_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.google.com
Registrar URL: https://domains.google.com
Updated Date: 2023-02-25T19:29:42Z
Creation Date: 2023-02-25T19:29:40Z
Registrar Registration Expiration Date: 2024-02-25T19:29:40Z
Registrar: Google LLC
Registrar IANA ID: 895
Registrar Abuse Contact Email: registrar-abuse@google.com
Registrar Abuse Contact Phone: +1.8772376466
Domain Status: clientTransferProhibited https://www.icann.org/epp#clientTransferProhibited
Registry Registrant ID:
Registrant Name: Contact Privacy Inc. Customer 7151571251
Registrant Organization: Contact Privacy Inc. Customer 7151571251
Registrant Street: 96 Mowat Ave
Registrant City: Toronto
Registrant State/Province: ON
Registrant Postal Code: M4K 3K1
Registrant Country: CA
Registrant Phone: +1.4165385487
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: https://domains.google.com/contactregistrant?domain=trygloo.com
Registry Admin ID:
Admin Name: Contact Privacy Inc. Customer 7151571251
Admin Organization: Contact Privacy Inc. Customer 7151571251
Admin Street: 96 Mowat Ave
Admin City: Toronto
Admin State/Province: ON
Admin Postal Code: M4K 3K1
Admin Country: CA
Admin Phone: +1.4165385487
Admin Phone Ext:
Admin Fax:
Admin Fax Ext:
Admin Email: https://domains.google.com/contactregistrant?domain=trygloo.com
Registry Tech ID:
Tech Name: Contact Privacy Inc. Customer 7151571251
Tech Organization: Contact Privacy Inc. Customer 7151571251
Tech Street: 96 Mowat Ave
Tech City: Toronto
Tech State/Province: ON
Tech Postal Code: M4K 3K1
Tech Country: CA
Tech Phone: +1.4165385487
Tech Phone Ext:
Tech Fax:
Tech Fax Ext:
Tech Email: https://domains.google.com/contactregistrant?domain=trygloo.com
Name Server: NS-CLOUD-B1.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-B2.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-B3.GOOGLEDOMAINS.COM
Name Server: NS-CLOUD-B4.GOOGLEDOMAINS.COM
DNSSEC: signedDelegation
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2023-07-12T21:37:25.975625Z <<<

For more information on Whois status codes, please visit
https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en

Please register your domains at: https://domains.google.com/
This data is provided by Google for information purposes, and to assist
persons obtaining information about or related to domain name registration
records. Google does not guarantee its accuracy.
By submitting a WHOIS query, you agree that you will use this data only for
lawful purposes and that, under no circumstances, will you use this data to:
1) allow, enable, or otherwise support the transmission of mass
   unsolicited, commercial advertising or solicitations via E-mail (spam); or
2) enable high volume, automated, electronic processes that apply to this
   WHOIS server.
These terms may be changed without prior notice.
By submitting this query, you agree to abide by this policy.
  • GET
    200 OK

    https://firefox.settings.services.mozilla.com/v1/buckets/main/collections/partitioning-exempt-urls/changeset?_expected=1688730865816

<html lang="en"><head><meta charset="utf-8"><link rel="stylesheet" href="/_next/static/css/e108ef4ee27e2c2e.css" data-precedence="next"><link rel="stylesheet" href="/_next/static/css/756b7389aa087689.css" data-precedence="next"><title>5 learnings from classifying 500k customer messages with LLMs vs traditional ML</title><meta name="description" content="Learnings from building text-classification for customer chat messages with AI."><meta name="viewport" content="width=device-width, initial-scale=1"><link rel="canonical" href="https://www.trygloo.com/blog/classify-text-llms-learnings"><link rel="icon" href="/favico.ico"><script src="/_next/static/chunks/polyfills-78c92fac7aa8fdd8.js" nomodule=""></script></head><body class="flex h-full min-h-screen w-screen  flex-col justify-center bg-gray-50"><div class="isolate h-full w-full bg-gray-50"><div class="absolute inset-x-0 top-[-10rem] -z-10 transform-gpu overflow-hidden blur-3xl sm:top-[-20rem]"><svg class="relative left-[calc(50%-11rem)] -z-10 h-[21.1875rem] max-w-none -translate-x-1/2 rotate-[30deg] sm:left-[calc(50%-30rem)] sm:h-[42.375rem]" viewBox="0 0 1155 678"><path fill="url(#f4773080-2a16-4ab4-9fd7-579fec69a4f7)" fill-opacity=".2" d="M317.219 518.975L203.852 678 0 438.341l317.219 80.634 204.172-286.402c1.307 132.337 45.083 346.658 209.733 145.248C936.936 126.058 882.053-94.234 1031.02 41.331c119.18 108.451 130.68 295.337 121.53 375.223L855 299l21.173 362.054-558.954-142.079z"></path><defs><linearGradient id="f4773080-2a16-4ab4-9fd7-579fec69a4f7" x1="1155.49" x2="-78.208" y1=".177" y2="474.645" gradientUnits="userSpaceOnUse"><stop stop-color="#9089FC"></stop><stop offset="1" stop-color="#FF80B5"></stop></linearGradient></defs></svg></div><div class="px-6 pt-6 lg:px-8"><nav class="flex items-center justify-between" aria-label="Global"><div class="flex lg:flex-1"><a href="#" class="-m-1.5 p-1.5"><span class="sr-only">Gloo</span><div class="flex flex-row items-center gap-x-1 font-semibold"><img alt="Gloo AI logo" loading="lazy" decoding="async" data-nimg="1" style="color:transparent" srcset="/_next/image?url=%2Fgloo-ai-square-256.png&amp;w=48&amp;q=75&amp;dpl=dpl_GG7eJeYaa8dyRd4NRdZQS4kSyMZo 1x, /_next/image?url=%2Fgloo-ai-square-256.png&amp;w=96&amp;q=75&amp;dpl=dpl_GG7eJeYaa8dyRd4NRdZQS4kSyMZo 2x" src="/_next/image?url=%2Fgloo-ai-square-256.png&amp;w=96&amp;q=75&amp;dpl=dpl_GG7eJeYaa8dyRd4NRdZQS4kSyMZo" width="48" height="48"><span class="text-lg">Gloo</span></div></a></div><div class="flex lg:hidden"><button type="button" class="-m-2.5 inline-flex items-center justify-center rounded-md p-2.5 text-gray-700"><span class="sr-only">Open main menu</span><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" class="h-6 w-6"><path fill-rule="evenodd" d="M2 4.75A.75.75 0 012.75 4h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 4.75zM2 10a.75.75 0 01.75-.75h14.5a.75.75 0 010 1.5H2.75A.75.75 0 012 10zm0 5.25a.75.75 0 01.75-.75h14.5a.75.75 0 010 1.5H2.75a.75.75 0 01-.75-.75z" clip-rule="evenodd"></path></svg></button></div><div class="hidden lg:flex lg:gap-x-12"><a href="/" class="text-base font-semibold leading-6 text-gray-700">Home</a><a href="/blog" class="text-base font-semibold leading-6 text-gray-700">Blog</a><a class="flex flex-row gap-x-2 text-base font-semibold leading-6 text-gray-700 hover:text-gray-500" href="/classify"><div>LLM Text Classification</div><div class=" animate-pulse border-purple-600 text-xs font-light text-purple-600">New!</div></a></div><div class="hidden lg:flex lg:flex-1 lg:justify-end"></div></nav><div style="position:fixed;top:1px;left:1px;width:1px;height:0;padding:0;margin:-1px;overflow:hidden;clip:rect(0, 0, 0, 0);white-space:nowrap;border-width:0;display:none"></div></div><article class="py-8"><div class="flex w-full flex-col items-center gap-y-8 px-8"><div><span class="font-light">Jun 24, 2023</span></div><article class="max-w-[800px]"><h1 class="whitespace-normal py-12 text-3xl font-semibold">5 learnings from classifying 500k customer messages with LLMs vs traditional ML</h1><div><div class="prose max-w-[1080px]"><p><a href="https://news.ycombinator.com/item?id=36681839">Join the discussion on Hacker News</a></p>
<p>LLMs can help you achieve state-of-the-art results on text classification tasks (like sentiment analysis, labeling, etc) just by describing the list of classes and descriptions.</p>
<p>We recently reached our 500k classification milestone using LLMs and fine-tuned models <span role="img" aria-label="party popper">🎉</span>, and below we share what we've learned.</p>
<h3 id="1-llms-prefer-outputting-something-over-nothing"><a aria-hidden="true" tabindex="-1" href="#1-llms-prefer-outputting-something-over-nothing"><span class="icon icon-link"></span></a><strong>1. LLMs prefer outputting something over nothing.</strong></h3>
<p>LLMs are trained to generate <em>any</em> text, so not generating anything is harder for them to do. This was a major cause for false-positives that we managed to correct once we added a catch-all class like "other" or "none-of-these".</p>
<h3 id="2-hallucinations-are-more-helpful-than-you-think"><a aria-hidden="true" tabindex="-1" href="#2-hallucinations-are-more-helpful-than-you-think"><span class="icon icon-link"></span></a><strong>2. Hallucinations are more helpful than you think</strong></h3>
<img src="/blog/classification-learnings/hallucination1.png" alt="Hallucination for LLM vs fine-tuned model">
<p>Keeping track of hallucinations can help you understand what good or bad class names are. In the example above you can see how the label "technical-issue" is hallucinated by the LLM, even though the actual label name in the prompt is "other/technical-issue".</p>
<p>Try to keep the class names as simple and direct as possible. Note that there may be other techniques that allow the LLM to not bias towards using the class name itself by replacing class names with symbols, called <a href="https://arxiv.org/pdf/2305.08298.pdf">symbol tuning</a> -- but this research is still ongoing.</p>
<h3 id="3-to-save-costs-and-improve-latency-fine-tuned-classification-models-are-the-answer-combining-them-with-llms-can-be-even-more-powerful"><a aria-hidden="true" tabindex="-1" href="#3-to-save-costs-and-improve-latency-fine-tuned-classification-models-are-the-answer-combining-them-with-llms-can-be-even-more-powerful"><span class="icon icon-link"></span></a><strong>3. To save costs and improve latency, fine tuned classification models are the answer. Combining them with LLMs can be even more powerful.</strong></h3>
<p>One customer needed to process some data at lower-latency than LLMs, and the only way we could do this was by forgoing ChatGPT usage altogether. To do this, we trained SBERT using contrastive learning on ChatGPT-labeled data to achieve around 85% parity with it on multi-label classification on the customer’s 22 different classes, and greater than 90% on a subset of those classes. The cost breakdown is below</p>
<img src="/blog/classification-learnings/bert-cost-openai.png" alt="Classification cost for bert vs chatgpt from openai.">
<h3 id="4-llm-reasoning-improves-accuracy"><a aria-hidden="true" tabindex="-1" href="#4-llm-reasoning-improves-accuracy"><span class="icon icon-link"></span></a><strong>4. LLM reasoning improves accuracy</strong></h3>
<p>This is a more well known prompt engineering technique, but it goes without saying — Chain of thought Prompting will result in higher accuracy when doing text classification ( <a href="https://arxiv.org/pdf/2305.08377.pdf">https://arxiv.org/pdf/2305.08377.pdf</a>). Prompting the LLM to extract a set of clues before doing the classification can yield state-of-the-art accuracy (96%+ on benchmarks). Below is an example from the paper</p>
<img src="/blog/classification-learnings/chainofthought1.png" alt="Chain of thought prompting">
<h3 id="5-standardizing-input-is-key-for-both-fine-tuned-models-and-llms"><a aria-hidden="true" tabindex="-1" href="#5-standardizing-input-is-key-for-both-fine-tuned-models-and-llms"><span class="icon icon-link"></span></a><strong>5. Standardizing input is key for both fine tuned models and LLMs.</strong></h3>
<p>The more text there is to classify, the less accurate the prediction.</p>
<p>Imagine you have a long running chat thread between you bot and a user. At every step, you’re attempting to determine the intent of the user. To improve prediction accuracy, we apply a preprocessing step to paraphrase the last user message with the previous context. This helps with classifying chat messages with multiple contexts, emails, long documents, and non-english messages.</p>
<img src="/blog/classification-learnings/normalization-diagram.png" alt="Summarizing text with ChatGPT before classifying it">
<hr>
<p>We built Gloo to help others solve text-classification problems in a more automated way. Gloo allows you to</p>
<ol>
<li>Create an LLM-classifier with prompt engineering, that is guaranteed to only output your specified classes.</li>
<li>Quantifiably measure how prompt changes impacts production (latency, accuracy, biases in class selection)</li>
<li>Train + deploy a traditional BERT based classifier based on your LLM data</li>
<li>Build a classifier that combines both the trained model + LLM for any new classes you didn’t train on</li>
</ol>
<img src="/blog/classification-learnings/gloo-classification-img1.png" alt="Gloo prompt engineering dashboard">
<p>** If you are interested in improving the accuracy, latency, or cost of your NLP text classifier, reach out to us: <a href="mailto:founders@trygloo.com">founders@trygloo.com</a> and we'll get you started with a free trial **</p></div></div></article><footer class="flex w-full items-center justify-center"><hr><p>Thanks for reading!</p></footer><div class="navigation_navigation__NQBJF"><div class="navigation_previous__zQj0E"></div><div class="navigation_next__zxAw7"></div></div></div></article></div><div class="mt-auto"><footer class=""><div class="mx-auto max-w-7xl py-12 px-6 md:flex md:items-center md:justify-between lg:px-8"><div class="flex justify-center space-x-6 md:order-2"><a href="#" class="text-gray-400 hover:text-gray-500"><span class="sr-only">GitHub</span><svg fill="currentColor" viewBox="0 0 24 24" class="h-6 w-6" aria-hidden="true"><path fill-rule="evenodd" d="M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z" clip-rule="evenodd"></path></svg></a></div><div class="mt-8 md:order-1 md:mt-0"><p class="text-center text-xs leading-5 text-gray-500">© 2023 Gloo Chat Inc. All rights reserved.</p><div class="flex flex-row gap-x-6"><p><a href="https://www.trygloo.com/privacy.html" class="text-center text-xs leading-5 text-gray-500">Privacy Policy</a></p><p><a href="https://www.trygloo.com/terms.html" class="text-center text-xs leading-5 text-gray-500">Terms of Use</a></p></div></div></div></footer></div><script src="/_next/static/chunks/webpack-0b26b437a9587559.js" async=""></script><script src="/_next/static/chunks/bce60fc1-1e2fdfa4c7a0614c.js" async=""></script><script src="/_next/static/chunks/769-766b9a2124af4d14.js" async=""></script><script src="/_next/static/chunks/main-app-6118df2562cf8b41.js" async=""></script><script>(self.__next_f=self.__next_f||[]).push([0])</script><script>self.__next_f.push([1,"1:HL[\"/_next/static/css/e108ef4ee27e2c2e.css\",{\"as\":\"style\"}]\n0:\"$L2\"\n"])</script><script>self.__next_f.push([1,"3:HL[\"/_next/static/css/756b7389aa087689.css\",{\"as\":\"style\"}]\n"])</script><script>self.__next_f.push([1,"4:I{\"id\":\"8802\",\"chunks\":[\"272:static/chunks/webpack-0b26b437a9587559.js\",\"253:static/chunks/bce60fc1-1e2fdfa4c7a0614c.js\",\"769:static/chunks/769-766b9a2124af4d14.js\"],\"name\":\"\",\"async\":false}\n6:I{\"id\":\"4299\",\"chunks\":[\"272:static/chunks/webpack-0b26b437a9587559.js\",\"253:static/chunks/bce60fc1-1e2fdfa4c7a0614c.js\",\"769:static/chunks/769-766b9a2124af4d14.js\"],\"name\":\"\",\"async\":false}\n8:I{\"id\":\"3211\",\"chunks\":[\"272:static/chunks/webpack-0b26b437a9587559.js\",\"253:static/chunks/bce60fc1-1e2fdfa4c7a0614c.js\",\"76"])</script><script>self.__next_f.push([1,"9:static/chunks/769-766b9a2124af4d14.js\"],\"name\":\"\",\"async\":false}\n9:I{\"id\":\"5767\",\"chunks\":[\"272:static/chunks/webpack-0b26b437a9587559.js\",\"253:static/chunks/bce60fc1-1e2fdfa4c7a0614c.js\",\"769:static/chunks/769-766b9a2124af4d14.js\"],\"name\":\"\",\"async\":false}\na:I{\"id\":\"5194\",\"chunks\":[\"962:static/chunks/962-94efc9d276edac2d.js\",\"477:static/chunks/477-8d74a46126de408f.js\",\"964:static/chunks/964-b93f77803436e56e.js\",\"768:static/chunks/app/blog/layout-790f14b0e583353a.js\"],\"name\":\"TopNavbar\",\"async\":false}\n"])</script><script>self.__next_f.push([1,"2:[[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/e108ef4ee27e2c2e.css\",\"precedence\":\"next\"}]],[\"$\",\"$L4\",null,{\"buildId\":\"n5Oj3RTcdD0NHJSYIHHWt\",\"assetPrefix\":\"\",\"initialCanonicalUrl\":\"/blog/classify-text-llms-learnings\",\"initialTree\":[\"\",{\"children\":[\"blog\",{\"children\":[[\"slug\",\"classify-text-llms-learnings\",\"d\"],{\"children\":[\"__PAGE__?{\\\"slug\\\":\\\"classify-text-llms-learnings\\\"}\",{}]}]}]},\"$undefined\",\"$undefined\",true],\"initialHead\":[\"$L5\",null],\"globalErrorComponent\":\"$6\",\"notFound\":[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[\"$\",\"body\",null,{\"className\":\"flex h-full min-h-screen w-screen  flex-col justify-center bg-gray-50\",\"children\":[[\"$\",\"div\",null,{\"className\":\"isolate h-full w-full bg-gray-50\",\"children\":[[\"$\",\"div\",null,{\"className\":\"absolute inset-x-0 top-[-10rem] -z-10 transform-gpu overflow-hidden blur-3xl sm:top-[-20rem]\",\"children\":[\"$\",\"svg\",null,{\"className\":\"relative left-[calc(50%-11rem)] -z-10 h-[21.1875rem] max-w-none -translate-x-1/2 rotate-[30deg] sm:left-[calc(50%-30rem)] sm:h-[42.375rem]\",\"viewBox\":\"0 0 1155 678\",\"children\":[[\"$\",\"path\",null,{\"fill\":\"url(#f4773080-2a16-4ab4-9fd7-579fec69a4f7)\",\"fillOpacity\":\".2\",\"d\":\"M317.219 518.975L203.852 678 0 438.341l317.219 80.634 204.172-286.402c1.307 132.337 45.083 346.658 209.733 145.248C936.936 126.058 882.053-94.234 1031.02 41.331c119.18 108.451 130.68 295.337 121.53 375.223L855 299l21.173 362.054-558.954-142.079z\"}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"linearGradient\",null,{\"id\":\"f4773080-2a16-4ab4-9fd7-579fec69a4f7\",\"x1\":\"1155.49\",\"x2\":\"-78.208\",\"y1\":\".177\",\"y2\":\"474.645\",\"gradientUnits\":\"userSpaceOnUse\",\"children\":[[\"$\",\"stop\",null,{\"stopColor\":\"#9089FC\"}],[\"$\",\"stop\",null,{\"offset\":1,\"stopColor\":\"#FF80B5\"}]]}]}]]}]}],[\"$L7\",\"$undefined\",[[\"$\",\"title\",null,{\"children\":\"404: This page could not be found.\"}],[\"$\",\"div\",null,{\"style\":{\"fontFamily\":\"system-ui,\\\"Segoe UI\\\",Roboto,Helvetica,Arial,sans-serif,\\\"Apple Color Emoji\\\",\\\"Segoe UI Emoji\\\"\",\"height\":\"100vh\",\"textAlign\":\"center\",\"display\":\"flex\",\"flexDirection\":\"column\",\"alignItems\":\"center\",\"justifyContent\":\"center\"},\"children\":[\"$\",\"div\",null,{\"children\":[[\"$\",\"style\",null,{\"dangerouslySetInnerHTML\":{\"__html\":\"body{color:#000;background:#fff;margin:0}.next-error-h1{border-right:1px solid rgba(0,0,0,.3)}@media (prefers-color-scheme:dark){body{color:#fff;background:#000}.next-error-h1{border-right:1px solid rgba(255,255,255,.3)}}\"}}],[\"$\",\"h1\",null,{\"className\":\"next-error-h1\",\"style\":{\"display\":\"inline-block\",\"margin\":\"0 20px 0 0\",\"padding\":\"0 23px 0 0\",\"fontSize\":24,\"fontWeight\":500,\"verticalAlign\":\"top\",\"lineHeight\":\"49px\"},\"children\":\"404\"}],[\"$\",\"div\",null,{\"style\":{\"display\":\"inline-block\"},\"children\":[\"$\",\"h2\",null,{\"style\":{\"fontSize\":14,\"fontWeight\":400,\"lineHeight\":\"49px\",\"margin\":0},\"children\":\"This page could not be found.\"}]}]]}]}]]]]}],[\"$\",\"div\",null,{\"className\":\"mt-auto\",\"children\":[\"$\",\"footer\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"mx-auto max-w-7xl py-12 px-6 md:flex md:items-center md:justify-between lg:px-8\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex justify-center space-x-6 md:order-2\",\"children\":[[\"$\",\"a\",\"GitHub\",{\"href\":\"#\",\"className\":\"text-gray-400 hover:text-gray-500\",\"children\":[[\"$\",\"span\",null,{\"className\":\"sr-only\",\"children\":\"GitHub\"}],[\"$\",\"svg\",null,{\"fill\":\"currentColor\",\"viewBox\":\"0 0 24 24\",\"className\":\"h-6 w-6\",\"aria-hidden\":\"true\",\"children\":[\"$\",\"path\",null,{\"fillRule\":\"evenodd\",\"d\":\"M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z\",\"clipRule\":\"evenodd\"}]}]]}]]}],[\"$\",\"div\",null,{\"className\":\"mt-8 md:order-1 md:mt-0\",\"children\":[[\"$\",\"p\",null,{\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"© 2023 Gloo Chat Inc. All rights reserved.\"}],[\"$\",\"div\",null,{\"className\":\"flex flex-row gap-x-6\",\"children\":[[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"href\":\"https://www.trygloo.com/privacy.html\",\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"Privacy Policy\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"href\":\"https://www.trygloo.com/terms.html\",\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"Terms of Use\"}]}]]}]]}]]}]}]}]]}]}],\"asNotFound\":false,\"children\":[[\"$\",\"html\",null,{\"lang\":\"en\",\"children\":[\"$\",\"body\",null,{\"className\":\"flex h-full min-h-screen w-screen  flex-col justify-center bg-gray-50\",\"children\":[[\"$\",\"div\",null,{\"className\":\"isolate h-full w-full bg-gray-50\",\"children\":[[\"$\",\"div\",null,{\"className\":\"absolute inset-x-0 top-[-10rem] -z-10 transform-gpu overflow-hidden blur-3xl sm:top-[-20rem]\",\"children\":[\"$\",\"svg\",null,{\"className\":\"relative left-[calc(50%-11rem)] -z-10 h-[21.1875rem] max-w-none -translate-x-1/2 rotate-[30deg] sm:left-[calc(50%-30rem)] sm:h-[42.375rem]\",\"viewBox\":\"0 0 1155 678\",\"children\":[[\"$\",\"path\",null,{\"fill\":\"url(#f4773080-2a16-4ab4-9fd7-579fec69a4f7)\",\"fillOpacity\":\".2\",\"d\":\"M317.219 518.975L203.852 678 0 438.341l317.219 80.634 204.172-286.402c1.307 132.337 45.083 346.658 209.733 145.248C936.936 126.058 882.053-94.234 1031.02 41.331c119.18 108.451 130.68 295.337 121.53 375.223L855 299l21.173 362.054-558.954-142.079z\"}],[\"$\",\"defs\",null,{\"children\":[\"$\",\"linearGradient\",null,{\"id\":\"f4773080-2a16-4ab4-9fd7-579fec69a4f7\",\"x1\":\"1155.49\",\"x2\":\"-78.208\",\"y1\":\".177\",\"y2\":\"474.645\",\"gradientUnits\":\"userSpaceOnUse\",\"children\":[[\"$\",\"stop\",null,{\"stopColor\":\"#9089FC\"}],[\"$\",\"stop\",null,{\"offset\":1,\"stopColor\":\"#FF80B5\"}]]}]}]]}]}],[\"$\",\"$L8\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"hasLoading\":false,\"template\":[\"$\",\"$L9\",null,{}],\"templateStyles\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"childProp\":{\"current\":[[[\"$\",\"$La\",null,{}],[\"$\",\"article\",null,{\"className\":\"py-8\",\"children\":[\"$\",\"$L8\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"hasLoading\":false,\"template\":[\"$\",\"$L9\",null,{}],\"templateStyles\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"childProp\":{\"current\":[\"$Lb\",null],\"segment\":[\"slug\",\"classify-text-llms-learnings\",\"d\"]},\"styles\":[[\"$\",\"link\",\"0\",{\"rel\":\"stylesheet\",\"href\":\"/_next/static/css/756b7389aa087689.css\",\"precedence\":\"next\"}]]}]}]],null],\"segment\":\"blog\"},\"styles\":[]}]]}],[\"$\",\"div\",null,{\"className\":\"mt-auto\",\"children\":[\"$\",\"footer\",null,{\"className\":\"\",\"children\":[\"$\",\"div\",null,{\"className\":\"mx-auto max-w-7xl py-12 px-6 md:flex md:items-center md:justify-between lg:px-8\",\"children\":[[\"$\",\"div\",null,{\"className\":\"flex justify-center space-x-6 md:order-2\",\"children\":[[\"$\",\"a\",\"GitHub\",{\"href\":\"#\",\"className\":\"text-gray-400 hover:text-gray-500\",\"children\":[[\"$\",\"span\",null,{\"className\":\"sr-only\",\"children\":\"GitHub\"}],[\"$\",\"svg\",null,{\"fill\":\"currentColor\",\"viewBox\":\"0 0 24 24\",\"className\":\"h-6 w-6\",\"aria-hidden\":\"true\",\"children\":[\"$\",\"path\",null,{\"fillRule\":\"evenodd\",\"d\":\"M12 2C6.477 2 2 6.484 2 12.017c0 4.425 2.865 8.18 6.839 9.504.5.092.682-.217.682-.483 0-.237-.008-.868-.013-1.703-2.782.605-3.369-1.343-3.369-1.343-.454-1.158-1.11-1.466-1.11-1.466-.908-.62.069-.608.069-.608 1.003.07 1.531 1.032 1.531 1.032.892 1.53 2.341 1.088 2.91.832.092-.647.35-1.088.636-1.338-2.22-.253-4.555-1.113-4.555-4.951 0-1.093.39-1.988 1.029-2.688-.103-.253-.446-1.272.098-2.65 0 0 .84-.27 2.75 1.026A9.564 9.564 0 0112 6.844c.85.004 1.705.115 2.504.337 1.909-1.296 2.747-1.027 2.747-1.027.546 1.379.202 2.398.1 2.651.64.7 1.028 1.595 1.028 2.688 0 3.848-2.339 4.695-4.566 4.943.359.309.678.92.678 1.855 0 1.338-.012 2.419-.012 2.747 0 .268.18.58.688.482A10.019 10.019 0 0022 12.017C22 6.484 17.522 2 12 2z\",\"clipRule\":\"evenodd\"}]}]]}]]}],[\"$\",\"div\",null,{\"className\":\"mt-8 md:order-1 md:mt-0\",\"children\":[[\"$\",\"p\",null,{\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"© 2023 Gloo Chat Inc. All rights reserved.\"}],[\"$\",\"div\",null,{\"className\":\"flex flex-row gap-x-6\",\"children\":[[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"href\":\"https://www.trygloo.com/privacy.html\",\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"Privacy Policy\"}]}],[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"href\":\"https://www.trygloo.com/terms.html\",\"className\":\"text-center text-xs leading-5 text-gray-500\",\"children\":\"Terms of Use\"}]}]]}]]}]]}]}]}]]}]}],null]}]]\n"])</script><script>self.__next_f.push([1,"b:[\"$\",\"div\",null,{\"className\":\"flex w-full flex-col items-center gap-y-8 px-8\",\"children\":[[\"$\",\"div\",null,{\"children\":[[\"$\",\"span\",null,{\"className\":\"font-light\",\"children\":\"Jun 24, 2023\"}],null]}],[\"$\",\"article\",null,{\"className\":\"max-w-[800px]\",\"children\":[[\"$\",\"h1\",null,{\"className\":\"whitespace-normal py-12 text-3xl font-semibold\",\"children\":\"5 learnings from classifying 500k customer messages with LLMs vs traditional ML\"}],[\"$\",\"$L8\",null,{\"parallelRouterKey\":\"children\",\"segmentPath\":[\"children\",\"blog\",\"children\",[\"slug\",\"classify-text-llms-learnings\",\"d\"],\"children\"],\"error\":\"$undefined\",\"errorStyles\":\"$undefined\",\"loading\":\"$undefined\",\"loadingStyles\":\"$undefined\",\"hasLoading\":false,\"template\":[\"$\",\"$L9\",null,{}],\"templateStyles\":\"$undefined\",\"notFound\":\"$undefined\",\"notFoundStyles\":\"$undefined\",\"childProp\":{\"current\":[\"$Lc\",null],\"segment\":\"__PAGE__?{\\\"slug\\\":\\\"classify-text-llms-learnings\\\"}\"},\"styles\":[]}]]}],[\"$\",\"footer\",null,{\"className\":\"flex w-full items-center justify-center\",\"children\":[[\"$\",\"hr\",null,{}],[\"$\",\"p\",null,{\"children\":\"Thanks for reading!\"}]]}],[\"$\",\"div\",null,{\"className\":\"navigation_navigation__NQBJF\",\"children\":[[\"$\",\"div\",null,{\"className\":\"navigation_previous__zQj0E\",\"children\":\"$undefined\"}],[\"$\",\"div\",null,{\"className\":\"navigation_next__zxAw7\",\"children\":\"$undefined\"}]]}]]}]\n"])</script><script>self.__next_f.push([1,"7:[[\"$\",\"meta\",\"0\",{\"charSet\":\"utf-8\"}],[\"$\",\"meta\",\"1\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}]]\n"])</script><script>self.__next_f.push([1,"5:[[\"$\",\"meta\",\"0\",{\"charSet\":\"utf-8\"}],[\"$\",\"title\",\"1\",{\"children\":\"5 learnings from classifying 500k customer messages with LLMs vs traditional ML\"}],[\"$\",\"meta\",\"2\",{\"name\":\"description\",\"content\":\"Learnings from building text-classification for customer chat messages with AI.\"}],[\"$\",\"meta\",\"3\",{\"name\":\"viewport\",\"content\":\"width=device-width, initial-scale=1\"}],[\"$\",\"link\",\"4\",{\"rel\":\"canonical\",\"href\":\"https://www.trygloo.com/blog/classify-text-llms-learnings\"}],[\"$\",\"link\",\"5\",{\"rel\":\"icon\",\"href\":\"/favico.ico\"}]]\n"])</script><script>self.__next_f.push([1,"c:[\"$\",\"div\",null,{\"children\":[\"$\",\"div\",null,{\"className\":\"prose max-w-[1080px]\",\"children\":\"$Ld\"}]}]\n"])</script><script>self.__next_f.push([1,"d:[[\"$\",\"p\",null,{\"children\":[\"$\",\"a\",null,{\"href\":\"https://news.ycombinator.com/item?id=36681839\",\"children\":\"Join the discussion on Hacker News\"}]}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"LLMs can help you achieve state-of-the-art results on text classification tasks (like sentiment analysis, labeling, etc) just by describing the list of classes and descriptions.\"}],\"\\n\",[\"$\",\"p\",null,{\"children\":[\"We recently reached our 500k classification milestone using LLMs and fine-tuned models \",[\"$\",\"span\",null,{\"role\":\"img\",\"aria-label\":\"party popper\",\"children\":\"🎉\"}],\", and below we share what we've learned.\"]}],\"\\n\",[\"$\",\"h3\",null,{\"id\":\"1-llms-prefer-outputting-something-over-nothing\",\"children\":[[\"$\",\"a\",null,{\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"href\":\"#1-llms-prefer-outputting-something-over-nothing\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],[\"$\",\"strong\",null,{\"children\":\"1. LLMs prefer outputting something over nothing.\"}]]}],\"\\n\",[\"$\",\"p\",null,{\"children\":[\"LLMs are trained to generate \",[\"$\",\"em\",null,{\"children\":\"any\"}],\" text, so not generating anything is harder for them to do. This was a major cause for false-positives that we managed to correct once we added a catch-all class like \\\"other\\\" or \\\"none-of-these\\\".\"]}],\"\\n\",[\"$\",\"h3\",null,{\"id\":\"2-hallucinations-are-more-helpful-than-you-think\",\"children\":[[\"$\",\"a\",null,{\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"href\":\"#2-hallucinations-are-more-helpful-than-you-think\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],[\"$\",\"strong\",null,{\"children\":\"2. Hallucinations are more helpful than you think\"}]]}],\"\\n\",[\"$\",\"img\",null,{\"src\":\"/blog/classification-learnings/hallucination1.png\",\"alt\":\"Hallucination for LLM vs fine-tuned model\"}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"Keeping track of hallucinations can help you understand what good or bad class names are. In the example above you can see how the label \\\"technical-issue\\\" is hallucinated by the LLM, even though the actual label name in the prompt is \\\"other/technical-issue\\\".\"}],\"\\n\",[\"$\",\"p\",null,{\"children\":[\"Try to keep the class names as simple and direct as possible. Note that there may be other techniques that allow the LLM to not bias towards using the class name itself by replacing class names with symbols, called \",[\"$\",\"a\",null,{\"href\":\"https://arxiv.org/pdf/2305.08298.pdf\",\"children\":\"symbol tuning\"}],\" -- but this research is still ongoing.\"]}],\"\\n\",[\"$\",\"h3\",null,{\"id\":\"3-to-save-costs-and-improve-latency-fine-tuned-classification-models-are-the-answer-combining-them-with-llms-can-be-even-more-powerful\",\"children\":[[\"$\",\"a\",null,{\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"href\":\"#3-to-save-costs-and-improve-latency-fine-tuned-classification-models-are-the-answer-combining-them-with-llms-can-be-even-more-powerful\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],[\"$\",\"strong\",null,{\"children\":\"3. To save costs and improve latency, fine tuned classification models are the answer. Combining them with LLMs can be even more powerful.\"}]]}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"One customer needed to process some data at lower-latency than LLMs, and the only way we could do this was by forgoing ChatGPT usage altogether. To do this, we trained SBERT using contrastive learning on ChatGPT-labeled data to achieve around 85% parity with it on multi-label classification on the customer’s 22 different classes, and greater than 90% on a subset of those classes. The cost breakdown is below\"}],\"\\n\",[\"$\",\"img\",null,{\"src\":\"/blog/classification-learnings/bert-cost-openai.png\",\"alt\":\"Classification cost for bert vs chatgpt from openai.\"}],\"\\n\",[\"$\",\"h3\",null,{\"id\":\"4-llm-reasoning-improves-accuracy\",\"children\":[[\"$\",\"a\",null,{\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"href\":\"#4-llm-reasoning-improves-accuracy\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],[\"$\",\"strong\",null,{\"children\":\"4. LLM reasoning improves accuracy\"}]]}],\"\\n\",[\"$\",\"p\",null,{\"children\":[\"This is a more well known prompt engineering technique, but it goes without saying — Chain of thought Prompting will result in higher accuracy when doing text classification ( \",[\"$\",\"a\",null,{\"href\":\"https://arxiv.org/pdf/2305.08377.pdf\",\"children\":\"https://arxiv.org/pdf/2305.08377.pdf\"}],\"). Prompting the LLM to extract a set of clues before doing the classification can yield state-of-the-art accuracy (96%+ on benchmarks). Below is an example from the paper\"]}],\"\\n\",[\"$\",\"img\",null,{\"src\":\"/blog/classification-learnings/chainofthought1.png\",\"alt\":\"Chain of thought prompting\"}],\"\\n\",[\"$\",\"h3\",null,{\"id\":\"5-standardizing-input-is-key-for-both-fine-tuned-models-and-llms\",\"children\":[[\"$\",\"a\",null,{\"aria-hidden\":\"true\",\"tabIndex\":\"-1\",\"href\":\"#5-standardizing-input-is-key-for-both-fine-tuned-models-and-llms\",\"children\":[\"$\",\"span\",null,{\"className\":\"icon icon-link\"}]}],[\"$\",\"strong\",null,{\"children\":\"5. Standardizing input is key for both fine tuned models and LLMs.\"}]]}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"The more text there is to classify, the less accurate the prediction.\"}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"Imagine you have a long running chat thread between you bot and a user. At every step, you’re attempting to determine the intent of the user. To improve prediction accuracy, we apply a preprocessing step to paraphrase the last user message with the previous context. This helps with classifying chat messages with multiple contexts, emails, long documents, and non-english messages.\"}],\"\\n\",[\"$\",\"img\",null,{\"src\":\"/blog/classification-learnings/normalization-diagram.png\",\"alt\":\"Summarizing text with ChatGPT before classifying it\"}],\"\\n\",[\"$\",\"hr\",null,{}],\"\\n\",[\"$\",\"p\",null,{\"children\":\"We built Gloo to help others solve text-classification problems in a more automated way. Gloo allows you to\"}],\"\\n\",[\"$\",\"ol\",null,{\"children\":[\"\\n\",[\"$\",\"li\",null,{\"children\":\"Create an LLM-classifier with prompt engineering, that is guaranteed to only output your specified classes.\"}],\"\\n\",[\"$\",\"li\",null,{\"children\":\"Quantifiably measure how prompt changes impacts production (latency, accuracy, biases in class selection)\"}],\"\\n\",[\"$\",\"li\",null,{\"children\":\"Train + deploy a traditional BERT based classifier based on your LLM data\"}],\"\\n\",[\"$\",\"li\",null,{\"children\":\"Build a classifier that combines both the trained model + LLM for any new classes you didn’t train on\"}],\"\\n\"]}],\"\\n\",[\"$\",\"img\",null,{\"src\":\"/blog/classification-learnings/gloo-classification-img1.png\",\"alt\":\"Gloo prompt engineering dashboard\"}],\"\\n\",[\"$\",\"p\",null,{\"children\":[\"** If you are interested in improving the accuracy, latency, or cost of your NLP text classifier, reach out to us: \",[\"$\",\"a\",null,{\"href\":\"mailto:founders@trygloo.com\",\"children\":\"founders@trygloo.com\"}],\" and we'll get you started with a free trial **\"]}]]\n"])</script><div id="headlessui-portal-root"><div data-headlessui-portal=""><div></div></div></div><next-route-announcer style="position: absolute;"></next-route-announcer></body></html>

Screenshot: