1. Introduction

Adding a new object class to an existing object detection system can be more costly than it initially appears. Although the final goal may simply be to detect an additional category, achieving reliable performance typically requires a new data collection, annotation, training, and evaluation process.

The main challenge is often not the model architecture itself, but the amount of labeled data required before training can even begin.

This report explores how prompt-based vision models can potentially reduce this dependency by generating initial annotations from raw images. Instead of manually creating every annotation from scratch, these models can be used to produce initial predictions that are subsequently reviewed and corrected by humans.

This approach introduces a different workflow in which the model can contribute not only to the final detection system, but also to the creation of the training data required to build that system.

2. The Cost of Adding a New Object Class

Consider an object detection system that is already deployed and performs well on a predefined set of object categories.

If a new object category needs to be added, the conventional approach generally requires collecting representative images of the new object and manually annotating them before the detector can be trained or fine-tuned.

A typical workflow can be represented as:

New Object Class → Data Collection → Data Annotation → Model Training/Fine-tuning → Evaluation

Each stage introduces additional effort.

2.1 Data Collection

A useful dataset needs to represent the conditions in which the detector will eventually operate. Images may need to cover different:

• Viewpoints

• Object scales

• Backgrounds

• Lighting conditions

• Environments

• Object poses and appearances

Obtaining sufficient representative data can therefore become a significant bottleneck.

2.2 Manual Annotation

Once the images have been collected, the objects need to be annotated.

For object detection, this commonly means manually creating bounding boxes around instances of the target object.

As the number of images and objects increases, annotation can become one of the most time-consuming parts of the dataset creation process.

This leads to an important consideration:

The cost of adding a new object class is not limited to training a new model. A significant portion of the effort may occur before training even begins.

3. Prompt-Based Vision Models

One potential approach for reducing the annotation burden is the use of prompt-based vision models.

Unlike conventional object detectors that are trained to recognize a predefined set of categories, prompt-based models can accept information describing what the system should look for.

The prompt can take different forms depending on the model. For example, it may be a textual description such as:

"Find all excavators in this image."

Alternatively, the model may be provided with a visual example of the target object.

This changes the interaction with the vision model from a fixed classification or detection problem into a more flexible, query-driven process.

4. Using Model Predictions for Data Annotation

The most interesting application of prompt-based models in this context is not necessarily replacing a production detector.

Instead, they can be used as part of the data creation and annotation pipeline.

A conventional annotation workflow may look like:

Raw Images → Manual Annotation → Training Data

With a prompt-based model, the workflow can instead become:

Raw Images → Prompt → Model Predictions → Human Verification → Training Data

In this workflow, the model generates an initial set of predictions. A human annotator can then review the results, correct incorrect predictions, and add missing objects.

The human is therefore no longer required to create every annotation from an empty image.

This can potentially reduce the amount of manual work required to construct a dataset for a previously unsupported object category.

5. Example: Prompt-Based Object Detection

The following examples demonstrate how a prompt-based vision model can be used to identify a previously unspecified object category from an image.

Example 1 — Text Prompt

Input Image:

Prompt:  Tree.

A single-word prompt is used to instruct the model to identify the target object in the image.

Model Output:

Figure 1. Example of object detection using a text prompt. The model is instructed to identify excavators in the input image.

Example 2 — Visual Prompt

Reference Object:

Model Output:

Figure 2. Example of object detection using a visual example of the target object.

6. From Model Predictions to Training Data

The generated predictions do not necessarily need to be treated as the final annotations.

Instead, they can serve as initial annotations or pre-annotations.

For example:

1. Raw images are collected.

2. A target object is specified through a prompt.

3. The prompt-based model generates object predictions.

4. A human reviews the predictions.

5. Incorrect predictions are corrected.

6. Missing objects are added.

7. The verified annotations are added to the training dataset.

8. A specialized detector can then be trained or fine-tuned using the resulting dataset.

This creates a feedback loop in which a general-purpose vision model assists in creating the data needed to train a more specialized model.

7. Potential Benefits

Using prompt-based models as part of the annotation workflow can provide several potential advantages.

Reduced Manual Annotation

Instead of drawing every bounding box manually, annotators can start from model-generated predictions and focus their effort on verification and correction.

More Flexible Object Discovery

Prompt-based models can allow users to query images for objects that were not necessarily part of a conventional detector's fixed training categories.

8. Limitations and Considerations

Prompt-based models do not eliminate the challenges associated with dataset creation.

Their predictions may contain false positives, missed objects, inaccurate localization, or inconsistent results across different images and conditions.

Consequently, model-generated annotations should not automatically be considered equivalent to manually verified ground truth.

The effectiveness of this workflow depends on factors such as model capability, object appearance, image quality, prompt design, and the requirements of the final detection system.

9. Prompt-Based Annotation: Coming Soon to Sairone

We are currently developing a new annotation capability that will allow users to generate initial object annotations using text prompts, visual prompts, or a combination of both. Instead of creating every annotation from scratch, annotators will be able to start with model-generated predictions and focus on reviewing and refining the results.

The new capability will also support TIFF images, making it suitable for annotation workflows involving high-resolution imagery and geospatial datasets.

This feature is currently under development and will be available on the Sairone annotation platform very soon.

Explore Sairone Annotation Platform

10. Conclusion

Adding a new object class to an existing detection system can require substantially more effort than simply modifying the model's class list.

A significant part of this effort comes from collecting representative data and creating high-quality annotations.

Prompt-based vision models offer an alternative way to approach this problem. By allowing a model to generate initial predictions from textual or visual prompts, they can become part of the dataset creation process rather than being used only for the final inference task.

The resulting workflow can be summarized as:

Raw Data → Prompt → Model Predictions → Human Verification → Training Data → Specialized Detector

This does not remove the need for high-quality data or human oversight. Instead, it aims to shift the annotation process from creating every annotation manually toward reviewing and refining model-generated annotations.

As promptable vision models continue to improve, this approach could become an increasingly useful component of workflows for introducing new object categories into computer vision systems.

References

[1] Liu, S., et al. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ArXiv:2303.05499, 2023.

[2] Cheng, T., et al. YOLO-World: Real-Time Open-Vocabulary Object Detection. ArXiv:2401.17270, 2024.

[3] Xiao, B., et al. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. arXiv:2311.06242, 2023.

[4] Ren, T., et al. "Grounding DINO 1.5: Advance the 'Edge' of Open-Set Object Detection." arXiv:2405.10300, 2024.

[5] Carion, N., et al. "SAM 3: Segment Anything with Concepts." arXiv:2511.16719, 2025.

[6] Bai, S., et al. "Qwen3-VL Technical Report." arXiv:2511.21631, 2025.