TokenStream::from_str does not return an error on invalid tokens

[canndrew]

I tried passing some text which contains invalid Rust tokens to TokenStream::from_str hoping that I could inspect the Err variant to get information on where the lex error is. Instead from_str prints directly to stderr and panics.

This method should probably give me the option to handle the error myself since I can always .unwrap() if I want to.

[alexcrichton]

Thanks for the report, do you have an example test case and/or input which shows this? Definitely sounds like a bug!

[canndrew]

Yes, that would probably help. In my case I narrowed it down to TokenStream::from_str("\\s").

[dtolnay]

I was not able to reproduce this.

use core::str::FromStr;
use proc_macro2::TokenStream;

fn main() {
    println!("{:?}", TokenStream::from_str("\\s"));
}

Err(LexError)

[dtolnay]

It looks like the compiler's proc_macro::TokenStream::from_str is what panics. Closing in favor of rust-lang/rust#58736.

[canndrew]

proc_macro2 fails for me too, at least when I run it inside a procedural macro.

[canndrew]

Cargo.toml

[package]
name = "repro"
version = "0.0.0"
edition = "2018"

[lib]
proc-macro = true

[dependencies]
proc-macro2 = "0.4.27"

src/lib.rs

extern crate proc_macro;

use proc_macro::TokenStream;
use std::str::FromStr;

#[proc_macro_derive(ParseTokenStream)]
pub fn derive(_input: TokenStream) -> TokenStream {
    println!("{:#?}", proc_macro2::TokenStream::from_str("\\s"));
    TokenStream::new()
}

src/main.rs

extern crate proc_macro;

use proc_macro::TokenStream;
use std::str::FromStr;

#[proc_macro_derive(ParseTokenStream)]
pub fn derive(_input: TokenStream) -> TokenStream {
    println!("{:#?}", proc_macro2::TokenStream::from_str("\\s"));
    TokenStream::new()
}

Output of cargo check:

    Checking repro v0.0.0 (/home/shum/rust-test)
error: unknown start of token: \
 --> src/main.rs:1:10
  |
1 | #[derive(repro::ParseTokenStream)]
  |          ^^^^^^^^^^^^^^^^^^^^^^^

error: proc-macro derive panicked
 --> src/main.rs:1:10
  |
1 | #[derive(repro::ParseTokenStream)]
  |          ^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 2 previous errors

error: Could not compile `repro`.

To learn more, run the command again with --verbose.

[dtolnay]

Thanks. When in a procedural macro, proc_macro2 functions call proc_macro so this is still best tracked in rust-lang/rust#58736.

[canndrew]

Okay. I figured that might be happening. Thanks!