CodeDom and GenerateInMemory memory leak

AMercer

I may have painted myself into a corner with GenerateInMemory=true.

My app need a custom user step. Users want to code (sort of - they are not
programmers) some refinements to a search procedure. They can concoct a
moderately complex boolean expression, and that suffices. I will take their
statements, add top and bottom text, and now I can do a CodeDom compile.

I generate an assembly via CodeDom with GenerateInMemory. I make a
delegate, and I can call my function, and all of this works fine. It runs at
good speed (no boxing of arguments), and that is what I need since I will be
looping over an array of data to be searched.

The problem is that I have a memory leak because of not unloading the
assembly. So, what I need is a new appdomain, and at this point I am stuck.
I don't know how to call CompileAssemblyFromSource() with
GenerateInMemory=True and put the generated assembly in a new appdomain. I
have no dll to load into the new appdomain.

Does anyone have any bright ideas? Working VS2005 VB source code (with the
leak) is included below. Every call to TestDynamicCode() adds two handles as
reported by the task manager. The call to CompileAssemblyFromSource() is
what produces the leak. What I think I want to do is make an appdomain, tell
codedom to put my compiler output assembly in the appdomain, do my
calculations, and then unload the appdomain. I don't see how to do it.

-------------------------

Public Module DynamicCode

Public Function SourceCode() As String
' user provides the commented lines, my code provides the rest
' the user lines coded herein are just for testing
Dim a() As String = { _
"Imports System.Math", _
"Public Class DynamicClass", _
" Public Shared Function DynamicFunction( _", _
" Byval s1 As String, _", _
" Byval s2 As String, _", _
" Byval i1 As Integer, _", _
" Byval i2 As Integer, _", _
" Byval d1 As Double, _", _
" Byval d2 As Double, _", _
" Byval d3 As Double, _", _
" Byval d4 As Double) _", _
" As Boolean", _
" Dim Qualified, b1, b2, b3, b4, b5, b6, b7, b8, b9 As Boolean", _
" Dim d As Double = Sqrt(d1)", _
" b1 = (d1>d2) ' user provides this and perhaps many others", _
" Qualified = b1 ' user provides this, the final result", _
" Return Qualified", _
" End Function", _
"End Class"}
Return Join(a, vbLf)
End Function

Public Delegate Function DynamicDelegate( _
ByVal s1 As String, _
ByVal s2 As String, _
ByVal i1 As Integer, _
ByVal i2 As Integer, _
ByVal d1 As Double, _
ByVal d2 As Double, _
ByVal d3 As Double, _
ByVal d4 As Double) _
As Boolean ' this delegate must match the signature in SourceCode() above

Public Sub TestDynamicCode() ' success is no exceptions and no assert
failures

' make params, the compiler parameters
Dim params As New CodeDom.Compiler.CompilerParameters
params.GenerateInMemory = True ' Assembly is created in memory
params.TreatWarningsAsErrors = False
params.WarningLevel = 4
params.CompilerOptions = "/optionexplicit /optionstrict /nowarn
/optimize-"
'Dim refs() As String = {"System.dll", "Microsoft.VisualBasic.dll"}
'params.ReferencedAssemblies.AddRange(refs)

' make results by compiling the source code
' CompileAssemblyFromSource leaks 2 handles and about 1k memory
' i think it is because of not unloading the assembly
' however, you can't unload an assembly, you can only unload an appdomain
' everything works ok except for this leak
Dim provider As New Microsoft.VisualBasic.VBCodeProvider
Dim results As CodeDom.Compiler.CompilerResults =
provider.CompileAssemblyFromSource(params, SourceCode)
provider.Dispose()

' report compilation errors
Dim errors As String = ""
For Each err As CodeDom.Compiler.CompilerError In results.Errors
Const f As String = "Line {0}, Col {1}: Error {2} - {3}"
errors &= [String].Format(f, err.Line, err.Column, err.ErrorNumber,
err.ErrorText) & vbLf
Next err
Debug.Assert(errors = "", errors)

' make DynamicFunctionDelegate so we can execute strongly typed and
without boxing
Dim DynamicClass As Type =
results.CompiledAssembly.GetType("DynamicClass")
Dim flags As Reflection.BindingFlags = Reflection.BindingFlags.Public Or
Reflection.BindingFlags.Static
Dim DynamicFunction As Reflection.MethodInfo =
DynamicClass.GetMethod("DynamicFunction", flags)
Dim DynamicFunctionDelegate As DynamicDelegate =
CType([Delegate].CreateDelegate(GetType(DynamicDelegate), Nothing,
DynamicFunction), DynamicDelegate)
Debug.Assert(Not DynamicFunctionDelegate Is Nothing)

' prove that it works, raise the loop limit to measure execution speed
For i As Integer = 1 To 100
Dim a As Double = Rnd(1)
Dim b As Double = Rnd(1)
Dim c As Double = 0
Dim bResult As Boolean = DynamicFunctionDelegate("", "", 1, 2, a, b,
c, c)
Debug.Assert(bResult = (a > b))
Next i

End Sub

End Module

Tom Shelton

On Oct 17, 8:01 am, AMercer <AMer...@discussions.microsoft.com> wrote:
> I may have painted myself into a corner with GenerateInMemory=true.
>
> My app need a custom user step. Users want to code (sort of - they are not
> programmers) some refinements to a search procedure. They can concoct a
> moderately complex boolean expression, and that suffices. I will take their
> statements, add top and bottom text, and now I can do a CodeDom compile.
>
> I generate an assembly via CodeDom with GenerateInMemory. I make a
> delegate, and I can call my function, and all of this works fine. It runs at
> good speed (no boxing of arguments), and that is what I need since I will be
> looping over an array of data to be searched.
>
> The problem is that I have a memory leak because of not unloading the
> assembly. So, what I need is a new appdomain, and at this point I am stuck.
> I don't know how to call CompileAssemblyFromSource() with
> GenerateInMemory=True and put the generated assembly in a new appdomain. I
> have no dll to load into the new appdomain.
>
> Does anyone have any bright ideas? Working VS2005 VB source code (with the
> leak) is included below. Every call to TestDynamicCode() adds two handles as
> reported by the task manager. The call to CompileAssemblyFromSource() is
> what produces the leak. What I think I want to do is make an appdomain, tell
> codedom to put my compiler output assembly in the appdomain, do my
> calculations, and then unload the appdomain. I don't see how to do it.
>
> -------------------------
>
> Public Module DynamicCode
>
> Public Function SourceCode() As String
> ' user provides the commented lines, my code provides the rest
> ' the user lines coded herein are just for testing
> Dim a() As String = { _
> "Imports System.Math", _
> "Public Class DynamicClass", _
> " Public Shared Function DynamicFunction( _", _
> " Byval s1 As String, _", _
> " Byval s2 As String, _", _
> " Byval i1 As Integer, _", _
> " Byval i2 As Integer, _", _
> " Byval d1 As Double, _", _
> " Byval d2 As Double, _", _
> " Byval d3 As Double, _", _
> " Byval d4 As Double) _", _
> " As Boolean", _
> " Dim Qualified, b1, b2, b3, b4, b5, b6, b7, b8, b9 As Boolean", _
> " Dim d As Double = Sqrt(d1)", _
> " b1 = (d1>d2) ' user provides this and perhaps many others", _
> " Qualified = b1 ' user provides this, the final result", _
> " Return Qualified", _
> " End Function", _
> "End Class"}
> Return Join(a, vbLf)
> End Function
>
> Public Delegate Function DynamicDelegate( _
> ByVal s1 As String, _
> ByVal s2 As String, _
> ByVal i1 As Integer, _
> ByVal i2 As Integer, _
> ByVal d1 As Double, _
> ByVal d2 As Double, _
> ByVal d3 As Double, _
> ByVal d4 As Double) _
> As Boolean ' this delegate must match the signature in SourceCode() above
>
> Public Sub TestDynamicCode() ' success is no exceptions and no assert
> failures
>
> ' make params, the compiler parameters
> Dim params As New CodeDom.Compiler.CompilerParameters
> params.GenerateInMemory = True ' Assembly is created in memory
> params.TreatWarningsAsErrors = False
> params.WarningLevel = 4
> params.CompilerOptions = "/optionexplicit /optionstrict /nowarn
> /optimize-"
> 'Dim refs() As String = {"System.dll", "Microsoft.VisualBasic.dll"}
> 'params.ReferencedAssemblies.AddRange(refs)
>
> ' make results by compiling the source code
> ' CompileAssemblyFromSource leaks 2 handles and about 1k memory
> ' i think it is because of not unloading the assembly
> ' however, you can't unload an assembly, you can only unload an appdomain
> ' everything works ok except for this leak
> Dim provider As New Microsoft.VisualBasic.VBCodeProvider
> Dim results As CodeDom.Compiler.CompilerResults =
> provider.CompileAssemblyFromSource(params, SourceCode)
> provider.Dispose()
>
> ' report compilation errors
> Dim errors As String = ""
> For Each err As CodeDom.Compiler.CompilerError In results.Errors
> Const f As String = "Line {0}, Col {1}: Error {2} - {3}"
> errors &= [String].Format(f, err.Line, err.Column, err.ErrorNumber,
> err.ErrorText) & vbLf
> Next err
> Debug.Assert(errors = "", errors)
>
> ' make DynamicFunctionDelegate so we can execute strongly typed and
> without boxing
> Dim DynamicClass As Type =
> results.CompiledAssembly.GetType("DynamicClass")
> Dim flags As Reflection.BindingFlags = Reflection.BindingFlags.Public Or
> Reflection.BindingFlags.Static
> Dim DynamicFunction As Reflection.MethodInfo =
> DynamicClass.GetMethod("DynamicFunction", flags)
> Dim DynamicFunctionDelegate As DynamicDelegate =
> CType([Delegate].CreateDelegate(GetType(DynamicDelegate), Nothing,
> DynamicFunction), DynamicDelegate)
> Debug.Assert(Not DynamicFunctionDelegate Is Nothing)
>
> ' prove that it works, raise the loop limit to measure execution speed
> For i As Integer = 1 To 100
> Dim a As Double = Rnd(1)
> Dim b As Double = Rnd(1)
> Dim c As Double = 0
> Dim bResult As Boolean = DynamicFunctionDelegate("", "", 1, 2, a, b,
> c, c)
> Debug.Assert(bResult = (a > b))
> Next i
>
> End Sub
>
> End Module

I don't have the time at the moment to work up a complete example, but
here are some general thoughts on what you are going to have to do to
accomplish your goal.

Since you have no way of specifying the appdomain for the in memory
assembly, then you're going to have to actually compile the assembly
in the target domain. While this may sound straight forward, it
isn't... The reason is that you're going to have to set up a method
to communicate with this domain with out actually referencing ANY
types that are in the dynamic assembly. Why? While, as soon as you
do, then it will also be loaded into your main appdomain, and you
won't be able to remove it from memory...

The trick is to create a class in a separate dll that inherits from
MarshalByRefObject that acts as the actual class loader. Then you can
do something like this in your code:

Dim compileDomain As AppDomain = AppDomain.CreateDomain
("CompileDomain")
Dim loader As ClassLoader = DirectCast
(compileDomain.CreateInstanceAndUnwrap("LoaderName space",
"LoaderNamespace.ClassLoader")

loader.DoYourWork("paramerters here")
AppDomain.Unload(compileDomain)

That's pretty rough, but the idea is that you only reference the class
loader type. Then, you create an instance of that in the new domain
(which is what the CreateInstanceAndUnwrap method does for you). At
this point all your calls ot the loader are then marshaled into the
new appdomain via remoting - which shouldn't cause you much of a
performance issue IF you can engineer your Loader class in such a way
that you don't have to make to many calls to it. In other words, keep
the interface chunky, not chatty.

--
Tom Shelton