RubberduckParser 2.0-异步解析

IRubberduckParser界面发生了重大变化，现在看起来像这样：

public interface IRubberduckParser
{
    RubberduckParserState State { get; }
    Task ParseAsync(VBComponent component, CancellationToken token);
    void Resolve(CancellationToken token);
}

这个新界面的作用不只是打破所有现有的单元测试和几乎所有的单个功能：它翻转事物并集中化解析器状态，现在可以在所有功能之间共享...，而不是由许多功能激活，它现在由一个按键挂钩激活，该挂钩可以捕获VBE中的按键并异步启动解析器任务-可以在解析器工作时禁用功能在后台运行，并在处于“就绪”状态时重新启用...不会因阻塞调用而烦扰用户。
这里是实现：

public class RubberduckParser : IRubberduckParser
{
    private readonly VBE _vbe;
    private readonly Logger _logger;

    public RubberduckParser(VBE vbe, RubberduckParserState state)
    {
        _vbe = vbe;
        _state = state;
        _logger = LogManager.GetCurrentClassLogger();
    }

    private readonly RubberduckParserState _state;
    public RubberduckParserState State { get { return _state; } }

    public async Task ParseAsync(VBComponent vbComponent, CancellationToken token)
    {
        var component = vbComponent;

        var parseTask = Task.Run(() => ParseInternal(component, token), token);

        try
        {
            await parseTask;
        }
        catch (SyntaxErrorException exception)
        {
            State.SetModuleState(component, ParserState.Error, exception);
        }
        catch (OperationCanceledException)
        {
            // no need to blow up
        } 
    }

    public void Resolve(CancellationToken token)
    {
        var options = new ParallelOptions { CancellationToken = token };
        Parallel.ForEach(_state.ParseTrees, options, kvp =>
        {
            token.ThrowIfCancellationRequested();
            ResolveReferences(kvp.Key, kvp.Value, token);
        });
    }

    private IEnumerable<CommentNode> ParseComments(QualifiedModuleName qualifiedName)
    {
        var code = qualifiedName.Component.CodeModule.Code();
        var commentBuilder = new StringBuilder();
        var continuing = false;

        var startLine = 0;
        var startColumn = 0;

        for (var i = 0; i < code.Length; i++)
        {
            var line = code[i];
            var index = 0;

            if (continuing || line.HasComment(out index))
            {
                startLine = continuing ? startLine : i;
                startColumn = continuing ? startColumn : index;

                var commentLength = line.Length - index;

                continuing = line.EndsWith("_");
                if (!continuing)
                {
                    commentBuilder.Append(line.Substring(index, commentLength).TrimStart());
                    var selection = new Selection(startLine + 1, startColumn + 1, i + 1, line.Length + 1);

                    var result = new CommentNode(commentBuilder.ToString(), new QualifiedSelection(qualifiedName, selection));
                    commentBuilder.Clear();

                    yield return result;
                }
                else
                {
                    // ignore line continuations in comment text:
                    commentBuilder.Append(line.Substring(index, commentLength).TrimStart());
                }
            }
        }
    }

    private void ParseInternal(VBComponent vbComponent, CancellationToken token)
    {
        _state.ClearDeclarations(vbComponent);
        State.SetModuleState(vbComponent, ParserState.Parsing);

        var qualifiedName = new QualifiedModuleName(vbComponent);
        _state.SetModuleComments(vbComponent, ParseComments(qualifiedName));

        var obsoleteCallsListener = new ObsoleteCallStatementListener();
        var obsoleteLetListener = new ObsoleteLetStatementListener();

        var listeners = new IParseTreeListener[]
        {
            obsoleteCallsListener,
            obsoleteLetListener
        };

        token.ThrowIfCancellationRequested();

        ITokenStream stream;
        var code = vbComponent.CodeModule.Lines();
        var tree = ParseInternal(code, listeners, out stream);

        token.ThrowIfCancellationRequested();
        _state.AddTokenStream(vbComponent, stream);
        _state.AddParseTree(vbComponent, tree);

        // cannot locate declarations in one pass *the way it's currently implemented*,
        // because the context in EnterSubStmt() doesn't *yet* have child nodes when the context enters.
        // so we need to EnterAmbiguousIdentifier() and evaluate the parent instead - this *might* work.
        var declarationsListener = new DeclarationSymbolsListener(qualifiedName, Accessibility.Implicit, vbComponent.Type, _state.Comments, token);

        token.ThrowIfCancellationRequested();
        declarationsListener.NewDeclaration += declarationsListener_NewDeclaration;
        declarationsListener.CreateModuleDeclarations();

        token.ThrowIfCancellationRequested();
        var walker = new ParseTreeWalker();
        walker.Walk(declarationsListener, tree);
        declarationsListener.NewDeclaration -= declarationsListener_NewDeclaration;

        _state.ObsoleteCallContexts = obsoleteCallsListener.Contexts.Select(context => new QualifiedContext(qualifiedName, context));
        _state.ObsoleteLetContexts = obsoleteLetListener.Contexts.Select(context => new QualifiedContext(qualifiedName, context));

        State.SetModuleState(vbComponent, ParserState.Parsed);
    }

    private IParseTree ParseInternal(string code, IEnumerable<IParseTreeListener> listeners, out ITokenStream outStream)
    {
        var input = new AntlrInputStream(code);
        var lexer = new VBALexer(input);
        var tokens = new CommonTokenStream(lexer);
        var parser = new VBAParser(tokens);

        parser.AddErrorListener(new ExceptionErrorListener());
        foreach (var listener in listeners)
        {
            parser.AddParseListener(listener);
        }

        outStream = tokens;
        return parser.startRule();
    }

    private void declarationsListener_NewDeclaration(object sender, DeclarationEventArgs e)
    {
         _state.AddDeclaration(e.Declaration);
    }

    private void ResolveReferences(VBComponent component, IParseTree tree, CancellationToken token)
    {
        if (_state.GetModuleState(component) != ParserState.Parsed)
        {
            return;
        }

        _state.SetModuleState(component, ParserState.Resolving);
        var declarations = _state.AllDeclarations;

        var resolver = new IdentifierReferenceResolver(new QualifiedModuleName(component), declarations);
        var listener = new IdentifierReferenceListener(resolver, token);
        var walker = new ParseTreeWalker();
        walker.Walk(listener, tree);

        _state.SetModuleState(component, ParserState.Ready);
    }

    private class ObsoleteCallStatementListener : VBABaseListener
    {
        private readonly IList<VBAParser.ExplicitCallStmtContext> _contexts = new List<VBAParser.ExplicitCallStmtContext>();
        public IEnumerable<VBAParser.ExplicitCallStmtContext> Contexts { get { return _contexts; } }

        public override void EnterExplicitCallStmt(VBAParser.ExplicitCallStmtContext context)
        {
            var procedureCall = context.eCS_ProcedureCall();
            if (procedureCall != null)
            {
                if (procedureCall.CALL() != null)
                {
                    _contexts.Add(context);
                    return;
                }
            }

            var memberCall = context.eCS_MemberProcedureCall();
            if (memberCall == null) return;
            if (memberCall.CALL() == null) return;
            _contexts.Add(context);
        }
    }

    private class ObsoleteLetStatementListener : VBABaseListener
    {
        private readonly IList<VBAParser.LetStmtContext> _contexts = new List<VBAParser.LetStmtContext>();
        public IEnumerable<VBAParser.LetStmtContext> Contexts { get { return _contexts; } }

        public override void EnterLetStmt(VBAParser.LetStmtContext context)
        {
            if (context.LET() != null)
            {
                _contexts.Add(context);
            }
        }
    }
}

我正在介绍一个集中的RubberduckParserState类，它替换了旧的笨拙的Declarations类型：

public class RubberduckParserState
{
    // keys are the declarations; values indicate whether a declaration is resolved.
    private readonly ConcurrentDictionary<Declaration, ResolutionState> _declarations =
        new ConcurrentDictionary<Declaration, ResolutionState>();

    private readonly ConcurrentDictionary<VBComponent, ITokenStream> _tokenStreams =
        new ConcurrentDictionary<VBComponent, ITokenStream>();

    private readonly ConcurrentDictionary<VBComponent, IParseTree> _parseTrees =
        new ConcurrentDictionary<VBComponent, IParseTree>();

    public event EventHandler StateChanged;

    private void OnStateChanged()
    {
        var handler = StateChanged;
        if (handler != null)
        {
            handler.Invoke(this, EventArgs.Empty);
        }
    }

    private readonly ConcurrentDictionary<VBComponent, ParserState> _moduleStates =
        new ConcurrentDictionary<VBComponent, ParserState>();

    private readonly ConcurrentDictionary<VBComponent, SyntaxErrorException> _moduleExceptions =
        new ConcurrentDictionary<VBComponent, SyntaxErrorException>();

    public void SetModuleState(VBComponent component, ParserState state, SyntaxErrorException parserError = null)
    {
        _moduleStates[component] = state;
        _moduleExceptions[component] = parserError;

        Status = _moduleStates.Values.Any(value => value == ParserState.Error)
            ? ParserState.Error
            : _moduleStates.Values.Any(value => value == ParserState.Parsing)
                ? ParserState.Parsing
                : _moduleStates.Values.Any(value => value == ParserState.Resolving)
                    ? ParserState.Resolving
                    : ParserState.Ready;

    }

    public ParserState GetModuleState(VBComponent component)
    {
        return _moduleStates[component];
    }

    private ParserState _status;
    public ParserState Status { get { return _status; } private set { if(_status != value) {_status = value; OnStateChanged();} } }

    private IEnumerable<QualifiedContext> _obsoleteCallContexts = new List<QualifiedContext>();

    /// <summary>
    /// Gets <see cref="ParserRuleContext"/> objects representing 'Call' statements in the parse tree.
    /// </summary>
    public IEnumerable<QualifiedContext> ObsoleteCallContexts
    {
        get { return _obsoleteCallContexts; }
        internal set { _obsoleteCallContexts = value; }
    }

    private IEnumerable<QualifiedContext> _obsoleteLetContexts = new List<QualifiedContext>();

    /// <summary>
    /// Gets <see cref="ParserRuleContext"/> objects representing explicit 'Let' statements in the parse tree.
    /// </summary>
    public IEnumerable<QualifiedContext> ObsoleteLetContexts
    {
        get { return _obsoleteLetContexts; }
        internal set { _obsoleteLetContexts = value; }
    }

    private readonly ConcurrentDictionary<VBComponent, IEnumerable<CommentNode>> _comments =
        new ConcurrentDictionary<VBComponent, IEnumerable<CommentNode>>();

    public IEnumerable<CommentNode> Comments
    {
        get 
        {
            return _comments.Values.SelectMany(comments => comments);
        }
    }

    public void SetModuleComments(VBComponent component, IEnumerable<CommentNode> comments)
    {
        _comments[component] = comments;
    }

    /// <summary>
    /// Gets a copy of the collected declarations.
    /// </summary>
    public IEnumerable<Declaration> AllDeclarations { get { return _declarations.Keys.ToList(); } }

    /// <summary>
    /// Adds the specified <see cref="Declaration"/> to the collection (replaces existing).
    /// </summary>
    public void AddDeclaration(Declaration declaration)
    {
        if (_declarations.TryAdd(declaration, ResolutionState.Unresolved))
        {
            return;
        }

        if (RemoveDeclaration(declaration))
        {
            _declarations.TryAdd(declaration, ResolutionState.Unresolved);
        }
    }

    public void ClearDeclarations(VBComponent component)
    {
        var declarations = _declarations.Keys.Where(k =>
            k.QualifiedName.QualifiedModuleName.Project == component.Collection.Parent
            && k.ComponentName == component.Name);

        foreach (var declaration in declarations)
        {
            ResolutionState state;
            _declarations.TryRemove(declaration, out state);
        }
    }

    public void AddTokenStream(VBComponent component, ITokenStream stream)
    {
        _tokenStreams[component] = stream;
    }

    public void AddParseTree(VBComponent component, IParseTree parseTree)
    {
        _parseTrees[component] = parseTree;
    }

    public IEnumerable<KeyValuePair<VBComponent, IParseTree>> ParseTrees { get { return _parseTrees; } }

    public TokenStreamRewriter GetRewriter(VBComponent component)
    {
        return new TokenStreamRewriter(_tokenStreams[component]);
    }

    /// <summary>
    /// Removes the specified <see cref="declaration"/> from the collection.
    /// </summary>
    /// <param name="declaration"></param>
    /// <returns>Returns true when successful.</returns>
    private bool RemoveDeclaration(Declaration declaration)
    {
        ResolutionState state;
        return _declarations.TryRemove(declaration, out state);
    }

    /// <summary>
    /// Ensures parser state accounts for built-in declarations.
    /// This method has no effect if built-in declarations have already been loaded.
    /// </summary>
    public void AddBuiltInDeclarations(IHostApplication hostApplication)
    {
        if (_declarations.Any(declaration => declaration.Key.IsBuiltIn))
        {
            return;
        }

        var builtInDeclarations = VbaStandardLib.Declarations;

        // cannot be strongly-typed here because of constraints on COM interop and generics in the inheritance hierarchy. </rant>
        if (hostApplication /*is ExcelApp*/ .ApplicationName == "Excel") 
        {
            builtInDeclarations = builtInDeclarations.Concat(ExcelObjectModel.Declarations);
        }

        foreach (var declaration in builtInDeclarations)
        {
            AddDeclaration(declaration);
        }
    }
}

其他上下文
解析器在启动时在App类中启动，如下所示：

public void Startup()
{
    CleanReloadConfig();

    _appMenus.Initialize();
    _appMenus.Localize();

    Task.Delay(1000).ContinueWith(t =>
    {
        _parser.State.AddBuiltInDeclarations(_vbe.HostApplication());
        ParseAll();
    });

    _hook.Attach();
}

1000ms延迟是为了允许VBE本身进行初始化，因此HostApplication()扩展方法不会返回null。 _hook是一个低级键盘钩子，也可以启动解析器，如下所示：

private async void _hook_KeyPressed(object sender, KeyHookEventArgs e)
{
    await ParseComponentAsync(e.Component);
}

private async Task ParseComponentAsync(VBComponent component, bool resolve = true)
{
    var tokenSource = RenewTokenSource(component);

    var token = tokenSource.Token;
    await _parser.ParseAsync(component, token);

    if (resolve && !token.IsCancellationRequested)
    {
        using (var source = new CancellationTokenSource())
        {
            _parser.Resolve(source.Token);
        }
    }
}

private CancellationTokenSource RenewTokenSource(VBComponent component)
{
    if (_tokenSources.ContainsKey(component))
    {
        CancellationTokenSource existingTokenSource;
        _tokenSources.TryRemove(component, out existingTokenSource);
        existingTokenSource.Cancel();
        existingTokenSource.Dispose();
    }

    var tokenSource = new CancellationTokenSource();
    _tokenSources[component] = tokenSource;
    return tokenSource;
}

到目前为止，唯一已知的问题是低级键钩子似乎在Office 2016主机中不起作用-但可以在Office 2010中/在我的机器上按预期工作。 GitHub上的完整代码（在撰写本文时，最新提交）。
欢迎任何反馈和改进。

这次@ Mat'sMug没有俏皮标题吗？：P

#1 楼

嵌套的三元条件丑陋。嵌套在三个深度中的三元条件式对三次方来说是丑陋的：

Status = _moduleStates.Values.Any(value => value == ParserState.Error)
    ? ParserState.Error
    : _moduleStates.Values.Any(value => value == ParserState.Parsing)
        ? ParserState.Parsing
        : _moduleStates.Values.Any(value => value == ParserState.Resolving)
            ? ParserState.Resolving
            : ParserState.Ready;

我将在这里使用if。 TopinFrassi已经解决了可能不必要地昂贵的问题。

您可以将这些if组合成一个条件：

if (procedureCall != null)
{
    if (procedureCall.CALL() != null)
    {
        _contexts.Add(context);
        return;
    }
}

您可以结合使用这些条件，并确保将括号放在身体上：

if (memberCall == null) return;
if (memberCall.CALL() == null) return;

因为您通常使用花括号，所以我猜这些是由于接受R＃建议而引起的。您可以通过以下方法打开设置中的括号：“选项”，然后选择“代码编辑”->“ C＃”->“格式样式”->“括号布局”，然后在“强制括号”组中更改设置。

此处存在一个错误，其中解析器在解析器完成之前启动了解析器，从而导致混乱的结果。解析器必须在启动解析器之前完成。这可以通过更改以下方法来实现：

public async Task ParseAsync(VBComponent vbComponent, CancellationToken token)
{
    var component = vbComponent;

    var parseTask = Task.Run(() => ParseInternal(component, token), token);

    try
    {
        await parseTask;
    }
    catch (SyntaxErrorException exception)
    {
        State.SetModuleState(component, ParserState.Error, exception);
    }
    catch (OperationCanceledException)
    {
        // no need to blow up
    } 
}

首先，我们最好检查取消请求。在涉及parseTask的两个语句之前添加此行：

token.ThrowIfCancellationRequested();

接下来，将await parseTask;语句更改为：

parseTask.Wait(token);

>现在不需要async修饰符。因为此方法是异步调用的，所以解析器仍将异步运行，但是解析器无法启动，直到解析任务完成。

#2 楼

您无需检查null的参数！当方法/构造函数为public时，您需要检查一下，因为假定任何人都可以重用您的代码。如果您的班级是internal, protected or private，我不介意，但是现在您确实有一个问题，即在不期望的时刻会抛出NullReferenceException。

您经常使用var。这还不错。但是我认为仅当您可以通过阅读代码来猜测类型时，才使用var对于可读性很有用。在这种情况下：

var code = qualifiedName.Component.CodeModule.Code();

我不知道code的类型是什么。而且它被大量使用。

此外，我也不知道Code()方法是您使用的还是外部DLL，但这不遵守命名约定。我认为应该是GetCode()或一个属性Code。除非确定，否则此方法调用将对某些内容进行编码。但这看起来不是这样。

此：

if (memberCall == null) return;
if (memberCall.CALL() == null) return;

可以更改为：if (memberCall?.CALL() == null) return;

而且，当我们到达那里时，我知道这个社区中的许多人都尽可能地使用方括号，在这种情况下。它有一天可能会阻止错误，但是是的，我认为它会使代码膨胀。现在是您的要求：)

我不知道性能在那段代码中是否很重要（我想是的），但这是

Status = _moduleStates.Values.Any(value => value == ParserState.Error)
    ? ParserState.Error
    : _moduleStates.Values.Any(value => value == ParserState.Parsing)
        ? ParserState.Parsing
        : _moduleStates.Values.Any(value => value == ParserState.Resolving)
            ? ParserState.Resolving
            : ParserState.Ready;

如果有很多Values，可能会引起问题。当一次就足够时，您将遍历该集合三次。据我了解，您需要检查所有集合是否存在解析错误，然后针对其他状态检查所有集合。

为什么不为此迭代构建算法一旦？（我认为该算法也应采用自己的方法）

这样的事情：

var state = ParserState.Ready;

for (var value in _moduleStates.Values)
{
    if(value == ParserState.Error) 
    {
        state = ParserState.Error;  
        break;
    }
    else if(value == ParserState.Parsing)
        state = ParserState.Parsing;
    else if(state != ParserState.Parsing && value == ParserState.Resolving)
        state = ParserState.Resolving;
}

return state;

我认为它会更快。
/>

\ $ \ begingroup \ $
我还不能在该项目中使用C＃6.0空条件运算符。不错的答案-我喜欢将“从单个模块状态获取整体解析器状态”内容抽象为自己的方法的想法。
\ $ \ endgroup \ $
– Mathieu Guindon♦
15年11月24日在16:42

\ $ \ begingroup \ $
CodeModule.Code（）是一个扩展方法，该方法返回一个字符串，该字符串表示VBComponent的CodeModule中的代码。我同意这不是一个理想的名字。
\ $ \ endgroup \ $
– Mathieu Guindon♦
2015年11月25日，下午1:07

编程黑洞网

RubberduckParser 2.0-异步解析

评论

#1 楼

#2 楼

评论