OpenAI GPT-4 vs. Groq Mistral-8x7B

    A few months previously, we ran a benchmark on an routine parser and Mistral 7B (Commence Supply LLM). The quality of the parsed result from Mistral 7B is reasonably impressive given it is splendid 7B parameters. One thing we weren’t happy with is the processing time. Only currently, I stumbled upon Groq who location the mission to revolutionize inference bound. They developed a chip for inference and to boot they called it the Language Processing Unit (LPU). I in actuality have examined it and it is in actuality impressive. I save no longer realize the technology of a chip however admire CPU and GPU, I imagine it could get faster and confidently, we can get the inference bound the general sort down to 1 2d consistently.

    Given the bound of inference has been brought down. I am to position a matter to if the usual is on par with OpenAI GPT-4 in parsing HTML. Be pleased the outdated article, we’re going to have the choice to avoid losing a the same benchmark.

    The Comparability

    Inquire of: Dentist

    Inquire of: Dentist (1st Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Dental Resource Management",
          "type": "Dentist",
          "phone": "(512) 462-2959",
          "address": "Austin, TX",
          "years_in_business": null,
          "hours": "Closed ⋅ Opens 6 AM",
          "rating": null,
          "reviews": "No reviews",
          "price": null,
          "description": null,
          "service_options": {},
          "is_operating": false
        }

    Processed in 6.28s 

            {
          "title": "Dental Resource Management",
          "type": "Dentist",
          "phone": "(512) 462-2959",
          "address": "Austin, TX",
          "years_in_business": null,
          "hours": "Opens 6 AM",
          "rating": null,
          "reviews": "No reviews",
          "price": null,
          "description": null,
          "service_options": {
            "in store shopping": null,
            "delivery": null
          },
          "is_operating": "Closed"
        }

    Processed in 0.91s

    Mistral almost nailed it, the recount thing missing is the Closed in the hours. Nonetheless, the inference time is splendid 0.91 seconds, impressive.

    Inquire of: Dentist (2nd Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Dr. Brendan Loehr",
          "type": "Dentist",
          "phone": "(512) 713-1099",
          "address": "201 W 5th St",
          "years_in_business": null,
          "hours": null,
          "rating": null,
          "reviews": "0",
          "price": null,
          "description": null,
          "service_options": {},
          "is_operating": true
        }

    Processed in 5.87s 

            {
          "title": "Dr. Brendan Loehr",
          "type": "Dentist",
          "phone": "(512) 713-1099",
          "address": "201 W 5th St",
          "years_in_business": null,
          "hours": null,
          "rating": null,
          "reviews": "No reviews",
          "price": null,
          "description": null,
          "service_options": {
            "in store shopping": false,
            "delivery": false
          },
          "is_operating": true
        }

    Processed in 0.86s

    Mistral rankings completely and completes the project in splendid 0.86 seconds.

    Inquire of: Dentist (third Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Tejas Patel, DDS",
          "type": "Cosmetic dentist",
          "phone": "(512) 476-2336",
          "address": "221 W 6th St #940, Austin, TX 78701",
          "years_in_business": null,
          "hours": "Closed ⋅ Opens 8 AM Mon",
          "rating": 4.9,
          "reviews": 275,
          "price": null,
          "description": "Best cosmetic Dentist in Austin.",
          "service_options": null,
          "is_operating": false
        }

    Processed in 9.82s 

            {
          "title": "Tejas Patel, DDS",
          "type": "Cosmetic dentist",
          "phone": "(512) 476-2336",
          "address": "221 W 6th St #940, Austin, TX 78701",
          "years_in_business": null,
          "hours": "Closed",
          "rating": 4.9,
          "reviews": 275,
          "price": null,
          "description": "Best cosmetic Dentist in Austin.",
          "service_options": {
            "in store shopping": null,
            "delivery": null
          },
          "is_operating": false
        }

    Processed in 1.06s

    Mistral made a mistake in the hours, the outlet time is missing this time.

    Inquire of: Mexican Restaurant

    Inquire of: Mexican Restaurant (1st Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Tacos provincia",
          "type": "Mexican",
          "phone": "Not Available",
          "address": "658 Red River St",
          "years_in_business": "Not Available",
          "hours": "Not Available",
          "rating": "5.0",
          "reviews": "12",
          "price": "$1–10",
          "description": "Not Available",
          "service_options": {
            "Dine-in": true,
            "Curbside pickup": true
          },
          "is_operating": "Not Available"
        }

    Processed in 8.75s 

            {
          "title": "Tacos provincia",
          "type": "Mexican",
          "phone": "N/A",
          "address": "658 Red River St",
          "years_in_business": "N/A",
          "hours": "N/A",
          "rating": 5.0,
          "reviews": 12,
          "price": "$1–10",
          "description": "N/A",
          "service_options": {
            "Dine-in": true,
            "Curbside pickup": true
          },
          "is_operating": "N/A"
        }

    Processed in 1.55s

    👍

    Inquire of: Mexican Restaurant (2nd Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Velvet Taco",
          "type": "Mexican",
          "phone": "N/A",
          "address": "522 Congress Ave. Suite 100",
          "years_in_business": "N/A",
          "hours": "Closed ⋅ Opens 11 AM",
          "rating": 4.6,
          "reviews": 1500,
          "price": "$10–20",
          "description": "N/A",
          "service_options": {
            "Dine-in": true,
            "Takeout": true,
            "No-contact delivery": true
          },
          "is_operating": false
        }

    Processed in 10.70s 

            {
          "title": "Velvet Taco",
          "type": "Mexican",
          "phone": null,
          "address": "522 Congress Ave. Suite 100",
          "years_in_business": null,
          "hours": "Opens 11 AM",
          "rating": 4.6,
          "reviews": 1500,
          "price": "$10–20",
          "description": null,
          "service_options": {
            "Dine-in": true,
            "Takeout": true,
            "No-contact delivery": true
          },
          "is_operating": false
        }

    Processed in 1.02s

    The appropriate incompatibility is in the hours where Closed is disregarded. Nonetheless, it does elaborate the is_operating accurately.

    Inquire of: Mexican Restaurant (third Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Iron Cactus Mexican Restaurant and Margarita Bar",
          "type": "Mexican",
          "phone": "Not Available",
          "address": "606 Trinity St",
          "years_in_business": "Not Available",
          "hours": "Closed ⋅ Opens 11 AM",
          "rating": "4.1",
          "reviews": "3.7K",
          "price": "$20–30",
          "description": "Buzzy spot with happy hour & tequilas",
          "service_options": {
            "Not Available": true
          },
          "is_operating": false
        }

    Processed in 8.36s 

            {
          "title": "Iron Cactus Mexican Restaurant and Margarita Bar",
          "type": "Mexican Restaurant",
          "phone": "Not Found",
          "address": "606 Trinity St",
          "years_in_business": "Not Found",
          "hours": "Closed - Opens 11 AM",
          "rating": 4.1,
          "reviews": 3.7,
          "price": "$20–30",
          "description": "Buzzy spot with happy hour & tequilas",
          "service_options": {
            "in_store_shopping": false,
            "delivery": false,
            "curbside_pickup": false
          },
          "is_operating": false
        }

    Processed in 1.22s

    3.7 need to collected be 3.7k or 3700 in Mistral, a severe mistake.

    Inquire of: Yoga Studio

    Inquire of: Yoga Studio (1st Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "YTX Yoga, Strength, & HIIT Training",
          "type": "Yoga studio",
          "phone": "(512) 502-5183",
          "address": "206 E 4th St, Austin, TX 78701",
          "years_in_business": "10+ years",
          "hours": "Closed ⋅ Opens 6:30 AM",
          "rating": "4.7",
          "reviews": "297",
          "price": null,
          "description": null,
          "service_options": {
            "Onsite services": true,
            "Online classes": true
          },
          "is_operating": false
        }

    Processed in 13.72s 

            {
          "title": "YTX Yoga, Strength, & HIIT Training",
          "type": "Yoga studio",
          "phone": "(512) 502-5183",
          "address": "206 E 4th St",
          "years_in_business": "10+",
          "hours": "Closed ⋅ Opens 6:30 AM",
          "rating": "4.7",
          "reviews": "297",
          "description": null,
          "service_options": {
            "Onsite services": true,
            "Online classes": true
          },
          "is_operating": false
        }

    Processed in 1.71s

    👍

    Inquire of: Yoga Studio (2nd Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Practice Yoga Austin",
          "type": "Yoga studio",
          "phone": "(512) 730-1638",
          "address": "1103 E 6th St, Austin, TX",
          "years_in_business": "10+ years",
          "hours": "Closed ⋅ Opens 7 AM",
          "rating": "5.0",
          "reviews": "97",
          "price": null,
          "description": ""Practice yoga studio is very unique."",
          "service_options": {},
          "is_operating": false
        }

    Processed in 10.52s 

            {
          "title": "Practice Yoga Austin",
          "type": "Yoga studio",
          "phone": "(512) 730-1638",
          "address": "1103 E 6th St",
          "years_in_business": "10+",
          "hours": "Closed - Opens 7 AM",
          "rating": "5.0",
          "reviews": "97",
          "description": "Practice yoga studio is very unique.",
          "service_options": {
            "in_store_shopping": false,
            "delivery": false,
            "curbside_pickup": false
          },
          "is_operating": false
        }

    Processed in 1.34s

    👍

    Inquire of: Yoga Studio (third Consequence)
    GPT-4 Mistral 8x7B 
            {
          "title": "Sanctuary Studio & Loft Rentals",
          "type": "Yoga studio",
          "phone": "(512) 337-2213",
          "address": "1006 S 8th St, Austin, TX 78704",
          "years_in_business": "10+ years",
          "hours": "Closed ⋅ Opens 9 AM",
          "rating": "4.8",
          "reviews": "55",
          "description": "Easygoing non-profit yoga center",
          "service_options": {},
          "is_operating": false
        }

    Processed in 9.40s 

            {
          "title": "Sanctuary Studio & Loft Rentals",
          "type": "Yoga studio",
          "phone": "(512) 337-2213",
          "address": "1006 S 8th St",
          "years_in_business": "10+ years",
          "hours": "Closed ⋅ Opens 9 AM",
          "rating": "4.8",
          "reviews": "55",
          "description": "Easygoing non-profit yoga center",
          "service_options": {
            "in_store_shopping": false,
            "delivery": false,
            "curbside_pickup": false
          },
          "is_operating": false
        }

    Processed in 1.74s

    👍

    Ideas

    GPT-4 scored a splendid salvage in parsing the HTML, however, the inference time isn’t any longer very finest. On the so a lot of hand, Mistral 8x7b runs on Groq does performs great faster; for about a of the outcomes it even goes under 1 2d. I deem it could even very nicely be regarded as for employ in the manufacturing, though it has made some errors in the discontinue result, this no longer no longer as a lot as could per chance even be resolved by bettering the suggested.

    I am taking a survey forward to experimenting more with Commence Supply LLM. Observe us to get our up-to-date sharing.

    Join us on X | YouTube

    Add a Characteristic Query💫 or a Bug🐞

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    You May Also Like