Need {% scripts %} in layout (https://docs.octobercms.com/3.x/markup/tag/placeholder.html#scripts)
{% framework extras %} if you use Flash Messages (https://docs.octobercms.com/3.x/cms/features/flash-messages.html)

Captcha Types

• default
• math
• flat
• mini
• inverse

To use your own settings, publish config.

$ php artisan vendor:publish --provider="Mews\Captcha\CaptchaServiceProvider"

config/captcha.php

return [
    'default'   => [
        'length'    => 5,
        'width'     => 120,
        'height'    => 36,
        'quality'   => 90,
        'math'      => true, //Enable Math Captcha
    ],
    // ...
];

Using example

url = "/captcha"
layout = "default"
title = "captcha"

[sitecaptcha]
type = "math"
showrefresh = "show"
iconclass = "icon-refresh"
==
<?php
function onSubmit()
{   
    $rules = ['captcha' => 'required|captcha_api:'. request('captcha_key') . ',math'];

    $messages = [
            'captcha.captcha_api' => 'wrong captcha',
        ];

    $validator = validator()->make(request()->all(), $rules, $messages);

    if ($validator->fails()) {

        throw new ValidationException($validator);

    } else {

        Flash::success('Success');

    }    

}
?>
==
<div class="container">
   <form data-request="onSubmit"  data-request-flash  data-request-complete="updateCaptcha(this)">
      <div class="mb-3">
         <label for="exampleInputEmail1" class="form-label">Login</label>
         <input type="text" name="login" class="form-control" id="exampleInputEmail1" aria-describedby="emailHelp">
         <div id="emailHelp" class="form-text">We ll never share your email with anyone else.</div>
      </div>
      <div class="mb-3">
         <label for="exampleInputPassword1" class="form-label">Password</label>
         <input type="password" name="pass" class="form-control" id="exampleInputPassword1">
      </div>
      <div class="mb-3">
         <div class="row">
            <div class="col-2">
               <label for="exampleInputPassword3" class="form-label">Captcha</label>
               <input type="text" name="captcha" class="form-control" id="exampleInputPassword3">
            </div>
            <div class="col-6 d-flex align-items-center mt-4 pt-2">
               {% component 'sitecaptcha' %}
            </div>
         </div>
      </div>
      <div class="mb-3 form-check">
         <input type="checkbox" class="form-check-input" id="exampleCheck1">
         <label class="form-check-label" for="exampleCheck1">Check me out</label>
      </div>
      <button type="submit" class="btn btn-primary">Submit</button>
   </form>
</div>

You can also use Captcha::check_api('captcha text', 'captcha key') to check captcha.

function onSubmit()
{   

    if (Captcha::check_api(request('captcha'), request('captcha_key'))){      
        Flash::success('Success');
    } else {
        Flash::error('Wrong captcha');
    }

}

Video manual