Issue 382/ support X-Robots-Tag as a typed http header XRobotsTag

[hafihaf123]

Initial Implementation of X-Robots-Tag Header

This pull request introduces an initial implementation of the X-Robots-Tag header for the rama-http crate and closes #382

Summary of Changes:

1. Header Registration:

   • Added the x-robots-tag header to the static headers list in rama-http-types/src/lib.rs.

2. Implementation of the Header:

   • Created a XRobotsTag struct that represents the header and implements the rama_http::headers::Header trait.
   • Defined a Rule enum to represent indexing rules such as noindex, nofollow, and max-snippet. Complex rules like max-image-preview and unavailable_after are also included.
   • Designed an Element struct to represent a rule optionally associated with a bot name.
   • Added an iterator for XRobotsTag to iterate over its elements.

3. File Structure:

   • The implementation resides in a new module at rama-http/src/headers/x_robots_tag/, which includes the following files:
     • rule.rs: Defines the Rule enum and parsing logic.
     • element.rs: Implements the Element struct and its parsing/formatting logic.
     • iterator.rs: Provides an iterator for XRobotsTag.
     • mod.rs: Combines and exposes the module’s functionality.

4. Encoding and Decoding:

   • Encoding and decoding are implemented using the Header trait, supporting CSV-style comma-separated values.

Questions and Feedback Requested:

1. Code Structure:

   • Is the location of the new module (x_robots_tag) appropriate?
   • Does the filename structure and organization align with project conventions?

2. Implementation Design:

   • Are there any improvements or alternative approaches you’d suggest for the Rule and Element structs?
   • Is the use of Vec<Element> for XRobotsTag suitable, or are there optimizations to consider?

3. Edge Cases and Standards:

   • Are there additional rules, formats, or edge cases that should be covered?

Testing:

• For now, the module is not tested, I will add the tests after the code structure becomes stable.

I look forward to your feedback and suggestions for improvement. Thank you!

[GlenDC]

Not a bad start. Do take your time to complete it and add sufficient start. But great start indeed. Added a couple of remarks to help you with some more guidance.

[GlenDC]

Suggested change

	static_header!["x-robots-tag",];
	//
	// More infornation at
	// <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Robots-Tag>.
	static_header!["x-robots-tag"];

[GlenDC]

You can copy https://github.com/hyperium/headers/blob/master/src/util/value_string.rs to https://github.com/plabayo/rama/tree/main/rama-http/src/headers/util and use that one here, that's also what the UserAgent typed header uses: https://docs.rs/headers/0.4.0/src/headers/common/user_agent.rs.html#43

[GlenDC]

Depending how you structure it, this actually has to be either:

struct Element { bot_name: Option<HeaderValueString>, rules: Vec<Rule> }

or

enum Element {
    BotName(HeaderValueString),
    Rule(Rule),
}

Because when a botname is mentioned it applies to all rules that follow it, until another botname is mentioned

[GlenDC]

You'll need to anyway modify the Element code, so this might dissapear, but as extra info, be careful with collecting stuff in a Vec, just for tmp reasons. Gives allocations for not much good. Instead better to split fixed into a tuple, or use the iterator directly .

[GlenDC]

This best be validated and enforced in a typed manner :)

[GlenDC]

Don't forget about the custom ones: noai, noimageai, SPC

[GlenDC]

Don't think it's worth implementing a public trait from this. You can just make simple private functions from this, and I would instead of collecting it into a vec just use the iterator's next method. You can then have a private fn for 1 arg en one with 2 arg.